REVIEW 4 major objections 6 minor 21 references
Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformers
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read CE-ViTs claims that averaging three ViT classifiers before conformal calibration lifts label coverage on mixed skin-lesion data to 90.38 percent, up from 80.43 for the best single model.
desk verdict Coverage comparison is uninterpretable because alpha is never reported; the paper is a routine application with a contradictory prediction-set definition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is an ensemble of three Vision Transformers whose softmax outputs are combined by simple averaging, followed by split conformal prediction. The conformal layer treats the averaged softmax score as a conformity score, estimates one quantile from a calibration set that pools HAM10000, Dermofit, and ISIC data, and builds for each test image a set of classes consistent with that quantile. The averaging step is what the paper credits with adapting to multiple domains; the conformal step is what converts the averaged scores into prediction sets with a stated coverage bound.
What would settle it
Calibrate on the pooled calibration split as the paper does, then evaluate coverage separately on each dataset's test split; if the per-source coverage on the most domain-shifted source falls far below the nominal level, the pooled coverage figure does not hold under the domain shift the paper claims to address.
Extended reading notes
Core claim
The central claim is that CE-ViTs improves conformal prediction through domain-adaptive ensembling rather than through better single-model accuracy. Each ViT is fine-tuned on a different skin dataset, and at inference the three softmax vectors are averaged; this averaged score is then fed to a conformal module that computes an empirical quantile from merged calibration data and outputs a prediction set per image. The paper reports coverage of 90.38 percent, exceeding the HAM10000 expert model's 80.43 percent, while increasing the average prediction set size for wrongly classified samples from 1.86 to 3.075 and producing low-to-intermediate uncertainty for correct predictions and high uncertainty for incorrect ones. The authors interpret these results as evidence that the ensemble's pooled prediction sets are more likely to contain the true lesion class under domain shift, and that overconfident misclassifications are reduced.
Load-bearing premise
The 90.38 percent coverage number is only a validity guarantee if the pooled test images and pooled calibration images come from the same mixture of sources and the seven lesion classes are labeled consistently across HAM10000, Dermofit, and ISIC; changing the source mixture or adding a new source would invalidate the guarantee.
Editorial extensions
If this is right
- If the pooled coverage claim holds, a skin-lesion screening tool can output a short list of plausible classes and tell the clinician that the true label falls in that list roughly 90 percent of the time.
- Merging calibration data from several sources before computing the quantile is a transferable recipe for conformal prediction under mixed domain shifts.
- The prediction-set pattern reported for CE-ViTs, small sets for correct cases and larger, higher-uncertainty sets for incorrect cases, is the behavior a decision-support system needs to avoid silent failures.
- Because the conformal layer only sees averaged scores, the same ensembling-with-calibration recipe could be applied to other classifier families, not only vision transformers.
Reading between the lines
- The paper reports only pooled coverage; a natural next step is to report coverage separately for each source, since a mixed test set can hide poor performance on one domain.
- The authors list weighted ensembles as future work; an implicit extension is tuning ensemble weights on calibration data to optimize set size at the same coverage level.
- The validity guarantee depends on exchangeability of pooled calibration and pooled test data, so deployment with different source proportions would need group-conditional or per-source calibration rather than one global quantile.
- A fair comparison of efficiency should hold coverage fixed: two methods with the same 90 percent target should be compared by average prediction set size, otherwise a larger set trivially buys more coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CE-ViTs, an ensemble of vision transformers trained on three skin-lesion datasets (HAM10000, Dermofit, ISIC) whose softmax outputs are averaged and then post-processed with split conformal prediction using a merged calibration set. The authors claim that CE-ViTs achieves 90.38% coverage on a pooled test set, a 9.95% improvement over the HAM10000 expert model, and that it produces larger prediction sets for misclassified samples (average size 3.075 vs. 1.86 for DMF). The paper also reports accuracy, precision, recall, and F1 metrics for the individual models and the ensemble.
Significance. If the claims were properly supported, combining ensemble learning with conformal prediction for domain-adaptive medical image classification would be a useful contribution. However, the evaluation as presented does not support the headline coverage claim: the target miscoverage rate alpha is never reported, the coverage values are compared without controlling for alpha, and the prediction-set definition in Eq. (1) contradicts the accompanying text. The paper also lacks error bars, confidence intervals, and a clear statement of the exchangeability assumptions for the merged calibration/test split. These are not presentation issues but invalidate the main empirical comparison. The paper does make use of a standard conformal-prediction framework, but the implementation and evaluation as described are not sufficient to establish the claimed improvements.
major comments (4)
- [Section II-C and Section IV] The target miscoverage rate alpha is never stated, although the conformal guarantee in Eq. (2) is explicitly conditioned on alpha. The headline result, 90.38% coverage versus 80.43% for the HAM10000 model, is therefore uninterpretable: if all models were calibrated at the same alpha on the same test distribution, their coverage values should all be close to 1-alpha up to finite-sample error. A 9.95 percentage-point gap indicates either different effective alpha values or invalid calibration, not improved uncertainty quantification. The authors must report alpha for every experiment, demonstrate that the same alpha was used, and show that calibration is valid on the pooled test distribution before any coverage comparison can be meaningful.
- [Section II-C, Eq. (1) and following text] Equation (1) defines the prediction set as C(X_test) = {y : S(X_test) < q_hat}, but the text immediately states that the set includes classes whose softmax output is less than 1 - q_hat. These definitions are contradictory. If the conformity score is the softmax output, the standard conformal set for classification is typically {y : S(X_test) >= 1 - q_hat} (or equivalently {y : S(X_test) > q_hat} with a different score definition). The implemented threshold therefore appears to be inconsistent with the stated formula, and this could directly affect all reported coverage values and set sizes.
- [Section III-A and III-B] The validity of the split conformal procedure requires that the calibration samples and test samples are exchangeable. The paper states that calibration data is a merged set of HAM10000, DMF, and ISIC and that the test set is 'combined,' but it does not specify how the pooled split was created—whether it was stratified by dataset, whether dataset proportions were preserved, or whether the split was random at the sample level. If the pooled test set is not exchangeable with the pooled calibration set, the reported coverage is not a valid guarantee. Moreover, per-dataset coverage is not reported, so the claimed domain-adaptation benefit over individual models is not substantiated.
- [Section IV, Tables 1–2 and Figures 3–4] All coverage and set-size results are reported as point estimates with no error bars, confidence intervals, or sample sizes. The finite-sample conformal bound in Eq. (2) has width 1/(n+1), so without n the reported 9.95% difference cannot be distinguished from finite-sample variation. The paper should report n, standard errors, or confidence intervals for coverage and set size, and should show whether the difference between CE-ViTs and the HAM10000 model is statistically significant.
minor comments (6)
- [Abstract] The phrase 'To aim this challenge' should read 'To address this challenge.'
- [Section II-C] The notation S = sigma f(X) is ambiguous; please clarify whether f denotes the logits and sigma the softmax, and define the conformity score explicitly.
- [Section IV, Table 1] The footnotes for Accuracy 1 and Accuracy 2 are confusing: Accuracy 1 is computed on each model's own training dataset, which is not a common test set. For a fair comparison, report accuracy on a single pooled test set for all models.
- [Section IV, Table 2] The symbols C_correct and C_incorrect are not defined in the text or table caption; please define them as average prediction-set sizes for correctly and incorrectly classified test samples.
- [Section IV, Figure 4] The 'uncertainty value' is never formally defined. Please specify how uncertainty is computed and what the histograms show.
- [Throughout] The dataset name is given as 'Dermofit' in the introduction and abstract but as 'DMF' in the methodology and tables; please use one consistent name.
Circularity Check
No significant circularity: the CE-ViTs pipeline is a standard ensemble-plus-split-conformal setup; the only self-citations are background and not load-bearing.
full rationale
The paper's derivation chain is not circular. CE-ViTs averages softmax outputs of three ViTs (Eq. 3) and applies split conformal prediction with a merged calibration set (Section III-B); the empirical coverage, set sizes, and uncertainty plots are measured on a held-out combined test set rather than being algebraic restatements of fitted parameters. The reported 90.38% coverage is consistent with Eq. (2) if alpha were near 0.10, but because alpha is never reported, one cannot exhibit the exact reduction of the coverage claim to a user-set parameter; this is a reporting/validity flaw, not a circular derivation. The two self-citations ([9], [11]) support only generic background statements about ViTs and conformal prediction; no load-bearing premise, uniqueness theorem, or ansatz is justified solely by these citations. The set-size results (Table II) are empirical and independent of any calibration target. No step in the paper fits a parameter to a subset and then relabels that fit as a prediction, and no known result is merely renamed. Accordingly, no circular step meeting the required evidence standard is present.
Assumptions & free parameters
free parameters (3)
- Conformal error rate alpha =
Not reported
- Learning rate =
0.0001
- Number of training epochs =
20
assumptions (3)
- domain assumption Calibration and test samples are exchangeable when drawn from the merged pool of HAM10000, Dermofit, and ISIC
- domain assumption The seven class labels (akiec, bcc, bkl, df, mel, nv, vasc) are semantically aligned across all three datasets
- domain assumption Softmax probabilities are used as conformity scores for conformal prediction
Cite this review
Pith. "Pith review of Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformers." pith.science (2026). https://pith.science/paper/F2JZJAGT
@misc{pith2026250515997,
author = {Pith},
title = {Pith review of: Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/F2JZJAGT}},
note = {Machine review of arXiv:2505.15997}
}
read the original abstract
Exploring the trustworthiness of deep learning models is crucial, especially in critical domains such as medical imaging decision support systems. Conformal prediction has emerged as a rigorous means of providing deep learning models with reliable uncertainty estimates and safety guarantees. However, conformal prediction results face challenges due to the backbone model's struggles in domain-shifted scenarios, such as variations in different sources. To aim this challenge, this paper proposes a novel framework termed Conformal Ensemble of Vision Transformers (CE-ViTs) designed to enhance image classification performance by prioritizing domain adaptation and model robustness, while accounting for uncertainty. The proposed method leverages an ensemble of vision transformer models in the backbone, trained on diverse datasets including HAM10000, Dermofit, and Skin Cancer ISIC datasets. This ensemble learning approach, calibrated through the combined mentioned datasets, aims to enhance domain adaptation through conformal learning. Experimental results underscore that the framework achieves a high coverage rate of 90.38\%, representing an improvement of 9.95\% compared to the HAM10000 model. This indicates a strong likelihood that the prediction set includes the true label compared to singular models. Ensemble learning in CE-ViTs significantly improves conformal prediction performance, increasing the average prediction set size for challenging misclassified samples from 1.86 to 3.075.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A survey on vision transformer,
K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu et al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence , vol. 45, no. 1, pp. 87–110, 2022
work page 2022
-
[2]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[3]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018
arXiv 2018
-
[4]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
work page 1901
-
[5]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[6]
Deepskin: a deep learning approach for skin cancer classification,
H. L. Gururaj, N. Manju, A. Nagarjun, V. M. Aradhya, and F. Flammini, “Deepskin: a deep learning approach for skin cancer classification,” IEEE Access, vol. 11, pp. 50 205–50 214, 2023
work page 2023
-
[7]
Skindistilvit: Lightweight vision transformer for skin lesion classification,
V.-C. Lungu-Stan, D.-C. Cercel, and F. Pop, “Skindistilvit: Lightweight vision transformer for skin lesion classification,” in International Confer- ence on Artificial Neural Networks . Springer, 2023, pp. 268–280
work page 2023
-
[8]
D. K. Saha, A. M. Joy, and A. Majumder, “Yotransvit: A transformer and cnn method for predicting and classifying skin diseases using segmentation techniques,” Informatics in Medicine Unlocked , vol. 47, p. 101495, 2024
work page 2024
Show all 21 references
-
[9]
Vision transformers in domain adaptation and generalization: A study of robustness,
S. Alijani, J. Fayyad, and H. Najjaran, “Vision transformers in domain adaptation and generalization: A study of robustness,” arXiv preprint arXiv:2404.04452, 2024
2024 arXiv
-
[10]
Domain adaptive ensemble learning,
K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain adaptive ensemble learning,” IEEE Transactions on Image Processing , vol. 30, pp. 8008– 8018, 2021
2021
-
[11]
Empirical validation of conformal prediction for trustworthy skin lesions classification,
J. Fayyad, S. Alijani, and H. Najjaran, “Empirical validation of conformal prediction for trustworthy skin lesions classification,” arXiv preprint arXiv:2312.07460, 2023
2023 arXiv
-
[12]
Swin transformer: Hierarchical vision transformer using shifted win- dows,
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted win- dows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022
2021
-
[13]
Making the best of both worlds: A domain-oriented transformer for unsupervised domain adaptation,
W. Ma, J. Zhang, S. Li, C. H. Liu, Y. Wang, and W. Li, “Making the best of both worlds: A domain-oriented transformer for unsupervised domain adaptation,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5620–5629
2022
-
[14]
Spectral unsupervised domain adaptation for visual recognition,
Y. Zhang, Q. Li, W. Wang, and Z. Zhang, “Spectral unsupervised domain adaptation for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022, pp. 12 345–12 354
2022
-
[15]
Cdtrans: Cross-domain transformer for unsupervised domain adaptation,
W. Li, L. Zhang, X. Wang, T. Liu, and D. Tao, “Cdtrans: Cross-domain transformer for unsupervised domain adaptation,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/pdf?id=XGzk5OKWFFc
2022
-
[16]
Empirically validating conformal prediction on modern vision architectures under distribution shift and long-tailed data,
K. Kasa and G. W. Taylor, “Empirically validating conformal prediction on modern vision architectures under distribution shift and long-tailed data,” arXiv preprint arXiv:2307.01088, 2023
2023 arXiv
-
[17]
Uncertainty sets for image classifiers using conformal prediction,
A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan, “Uncertainty sets for image classifiers using conformal prediction,” arXiv preprint arXiv:2009.14193, 2020
2009 arXiv
-
[18]
A gentle introduction to conformal prediction and distribution-free uncertainty quantification,
A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,”arXiv preprint arXiv:2107.07511, 2021
2021 arXiv
-
[19]
Skin cancer mnist: Ham10000,
“Skin cancer mnist: Ham10000,” https://www.kaggle.com/datasets/ kmader/skin-cancer-mnist-ham10000, accessed on July 11, 2024
2024
-
[20]
Skin cancer isic,
“Skin cancer isic,” https://www.kaggle.com/datasets/nodoubttome/ skin-cancer9-classesisic, accessed on July 11, 2024
2024
-
[21]
Pre-trained model,
“Pre-trained model,” https://www.mathworks.com/help/deeplearning/ug/ train-vision-transformer-network-for-image-classification.html, accessed on May 2, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.