REVIEW 3 major objections 4 minor 27 references
Automated Multi-label Classification of Eleven Retinal Diseases: A Benchmark of Modern Architectures and a Meta-Ensemble on a Large Synthetic Dataset
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Models trained only on synthetic fundus images classify eleven retinal diseases and generalize to real clinical scans, with external AUCs from 0.80 to 0.91.
desk verdict Useful benchmark on a new synthetic fundus dataset, but the 'exclusively synthetic' claim is undercut by RETFound's real-data pretraining. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pairing of a large synthetic fundus dataset, SynFundus-1M, with a stacking meta-ensemble: six diverse backbone architectures are trained per fold, their out-of-fold predictions are saved, and a gradient-boosted decision-tree model, XGBoost, learns to combine them. The ensemble is what smooths over each architecture's blind spots and produces the reported AUCs; the synthetic dataset is what removes the need for human-annotated clinical examples during training.
What would settle it
Train the identical six-architecture stacking pipeline on a real-image dataset covering the same eleven diseases, then test both the synthetic-trained and real-trained models on a held-out real clinical test set; if the synthetic-trained model's per-disease AUC falls substantially below the real-trained model's on the same test set, the claim that synthetic training alone suffices is falsified. A simpler check is per-disease external AUC on RFMiD: any disease whose external AUC drops near chance indicates a synthetic rendering gap.
Extended reading notes
Core claim
The paper's central claim is that a model trained exclusively on synthetic fundus photographs can accurately classify eleven retinal diseases and generalize to real clinical images. In concrete terms, the authors train ConvNeXtV2, SwinV2, ViT, ResNet, EfficientNetV2, and RETFound under a 5-fold multi-label stratified cross-validation scheme on SynFundus-1M, then stack the out-of-fold predictions with an XGBoost classifier to form a meta-ensemble. On the internal validation split, the ensemble attains a macro-average AUC of 0.9973. On external real-world data, it attains AUC 0.7972 on a combined diabetic retinopathy dataset, AUC 0.9126 on AIROGS, and macro-AUC 0.8800 on RFMiD, which the paper
Load-bearing premise
The central claim depends on SynFundus-1M's synthetic fundus images faithfully reproducing the visual features, disease morphologies, and imaging variability of real clinical fundus photographs; if the synthetic images are too clean or idealized, the reported external AUCs will not hold in broader clinical settings.
Editorial extensions
If this is right
- A model trained with zero real clinical images can serve as a baseline or starting point for retinal disease screening in settings where annotated data are scarce.
- The stacked ensemble's large internal-validation margin over individual architectures suggests that combining diverse backbones is a cheap way to improve robustness on this task.
- External results on three separate clinical datasets imply synthetic pretraining could reduce the cost and privacy burden of building ophthalmology AI systems.
- The benchmark gives future work a concrete reference point: new methods on SynFundus-1M can be compared against the six-architecture stacking baseline.
Reading between the lines
- An untested corollary is that per-disease transfer quality will vary with how faithfully SynFundus renders each pathology; reporting per-class external AUC would reveal which diseases still need real images.
- Because the external datasets cover diabetic retinopathy, glaucoma, and a multi-label retinal disease set, the generalization claim is demonstrated for those distributions; extending to other cameras, populations, and disease mixes is a natural stress test.
- If the synthetic-to-real transfer holds, similar large synthetic datasets could be generated for other organ systems where annotated medical images are scarce, but that extension is beyond what this paper shows.
- The comparison between RETFound, a foundation model pretrained on real images, and architectures trained from scratch on synthetic data could clarify how much of the transfer comes from synthetic data versus pretraining, but the paper does not isolate this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks six modern architectures (ConvNeXtV2, SwinV2, ViT, ResNet, EfficientNetV2, RETFound) on SynFundus-1M, a synthetic fundus dataset of over one million images labeled for eleven retinal diseases. Models are trained with 5-fold multi-label stratified cross-validation, and an XGBoost meta-ensemble is built on out-of-fold predictions. The meta-ensemble is reported to achieve an internal macro-AUC of 0.9973 and external AUCs of 0.7972 on a combined DR dataset, 0.9126 on AIROGS, and 0.8800 on RFMiD. The central claim is that models trained exclusively on synthetic data can accurately classify multiple pathologies and generalize to real clinical images.
Significance. If the central claim holds, the paper would provide a valuable benchmark for the large synthetic fundus dataset and a tractable pathway for developing multi-label retinal classifiers without large annotated clinical corpora. The study's strengths include a one-million-image synthetic training set, multiple modern architectures, 5-fold cross-validation, out-of-fold stacking, and external evaluation on three independently labeled real-world datasets. However, the 'exclusively synthetic' claim is currently under-supported because RETFound is pretrained on real retinal images, and because the internal AUC is measured on the synthetic validation distribution. The external results are suggestive but require a cleaner attribution and a clear definition of the combined DR dataset.
major comments (3)
- [Abstract / Architectures (RETFound)] The abstract claims models were 'trained exclusively on synthetic data,' but the architecture list includes RETFound, a foundation model pretrained on roughly 1.6M real retinal photographs. If RETFound is fine-tuned from those weights, the external AUCs on AIROGS, RFMiD, and the combined DR dataset cannot be cleanly attributed to synthetic-only training; they may reflect transfer of real-image representations. This is load-bearing because it is precisely the distinction between 'synthetic data fidelity drives generalization' and 'real-data pretraining carries the transfer.' Please either add an ablation that initializes the same architectures from random or ImageNet-only weights, or reword the claim to 'models fine-tuned on synthetic data from a real-pretrained foundation model.'
- [External validation / Results] The 'combined DR dataset' is undefined: no source, number of images, disease composition, labeling protocol, or split is provided. The reported external AUC of 0.7972 also has no confidence interval or error bar, which is especially concerning given it is substantially lower than the internal 0.9973. Without these details, the key generalization result is not reproducible. Please report the construction of each external dataset, sample sizes, confidence intervals (e.g., DeLong or bootstrap) for all external AUCs, and explicitly state that the internal 0.9973 is computed on the synthetic validation split, not on real clinical images.
- [Methods / Meta-ensemble] The meta-ensemble uses out-of-fold predictions and XGBoost, but the manuscript does not specify the XGBoost hyperparameters, the cross-validation scheme used to tune them, or whether the internal validation set was reused for any model selection beyond the reported stacking. Since the meta-ensemble is presented as the best-performing model, this information is needed to assess overfitting and to make the benchmark reproducible.
minor comments (4)
- [Full text / formatting] The supplied full text is heavily garbled (mojibake), making equations, tables, and several methods passages unreadable. If this reflects the submitted PDF rather than a review-system artifact, the authors must provide a cleanly rendered manuscript.
- [Abstract / Notation] Define 'DR' at first use in the abstract or introduction; 'combined DR dataset' is ambiguous without a citation or description.
- [Results / Per-class metrics] For the multi-label external datasets, report per-class AUCs and prevalence in addition to macro-AUC. Macro-AUC alone can obscure classes with poor performance, which is important for the 'comprehensive AI systems' claim.
- [Methods / Training details] Specify image resolution, augmentation schedule, optimization hyperparameters, and training epochs for each architecture. The reproducibility of the benchmark depends on these details, and they are not currently readable.
Circularity Check
No circularity: external benchmarks are independent of synthetic training data; the RETFound caveat is an attribution concern, not a circular derivation.
full rationale
This paper is an empirical benchmark and does not contain a derivation chain in which a predicted quantity is constructed from the same data used to define it. Base models are trained on SynFundus-1M under 5-fold stratified cross-validation; the internal validation macro-AUC is measured on held-out synthetic folds, and the external AUCs on AIROGS, RFMiD, and the combined DR dataset come from independent clinical datasets that were not used for fitting. The meta-ensemble uses XGBoost stacked on out-of-fold predictions, which is a standard cross-validation stacking scheme; even if the internal ensemble score were optimistic, the external numbers do not reduce to the training objective. The only notable issue is the abstract's phrase 'models trained exclusively on synthetic data' alongside the RETFound foundation model, which was pretrained on real retinal images. This is a potential attribution confound requiring an ablation from random or ImageNet-only initialization, but it is not circularity: it does not make the external AUCs equivalent to the training data by definition, nor does it rely on a self-citation chain. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation steps were found.
Assumptions & free parameters
free parameters (2)
- Meta-ensemble (XGBoost) parameters =
unknown
- Base model training hyperparameters =
unknown
assumptions (3)
- domain assumption SynFundus-1M synthetic images realistically represent the appearance of real fundus images and their disease features.
- domain assumption The eleven disease labels in the synthetic dataset are semantically aligned with the labels in the external real datasets (AIROGS, RFMiD, DR).
- domain assumption Macro-average AUC is a valid and sufficient metric for the multi-label class-imbalanced setting.
Cite this review
Pith. "Pith review of Automated Multi-label Classification of Eleven Retinal Diseases: A Benchmark of Modern Architectures and a Meta-Ensemble on a Large Synthetic Dataset." pith.science (2026). https://pith.science/paper/P2WZNRAS
@misc{pith2026250815986,
author = {Pith},
title = {Pith review of: Automated Multi-label Classification of Eleven Retinal Diseases: A Benchmark of Modern Architectures and a Meta-Ensemble on a Large Synthetic Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2WZNRAS}},
note = {Machine review of arXiv:2508.15986}
}
read the original abstract
The development of multi-label deep learning models for retinal disease classification is often hindered by the scarcity of large, expertly annotated clinical datasets due to patient privacy concerns and high costs. The recent release of SynFundus-1M, a high-fidelity synthetic dataset with over one million fundus images, presents a novel opportunity to overcome these barriers. To establish a foundational performance benchmark for this new resource, we developed an end-to-end deep learning pipeline, training six modern architectures (ConvNeXtV2, SwinV2, ViT, ResNet, EfficientNetV2, and the RETFound foundation model) to classify eleven retinal diseases using a 5-fold multi-label stratified cross-validation strategy. We further developed a meta-ensemble model by stacking the out-of-fold predictions with an XGBoost classifier. Our final ensemble model achieved the highest performance on the internal validation set, with a macro-average Area Under the Receiver Operating Characteristic Curve (AUC) of 0.9973. Critically, the models demonstrated strong generalization to three diverse, real-world clinical datasets, achieving an AUC of 0.7972 on a combined DR dataset, an AUC of 0.9126 on the AIROGS glaucoma dataset and a macro-AUC of 0.8800 on the multi-label RFMiD dataset. This work provides a robust baseline for future research on large-scale synthetic datasets and establishes that models trained exclusively on synthetic data can accurately classify multiple pathologies and generalize effectively to real clinical images, offering a viable pathway to accelerate the development of comprehensive AI systems in ophthalmology.
Reference graph
Works this paper leans on
-
[1]
S. R. Flaxman, R. R. Bourne, S. Resnikoff, P. Ackland, T. Braithwaite, M. V. Cicinelli, A. Das, J. B. Jonas, J. Keeffe, J. H. Kempen, et al., Global causes of blindness and distance vision impairment 1990--2020: a systematic review and meta-analysis, The Lancet Global Health 5 (12) (2017) e1221--e1234
work page 1990
-
[2]
R. R. Bourne, S. R. Flaxman, T. Braithwaite, M. V. Cicinelli, A. Das, J. B. Jonas, J. Keeffe, J. H. Kempen, J. Leasher, H. Limburg, et al., Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis, The Lancet Global Health 5 (9) (2017) e888--e897
work page 2017
-
[3]
V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, et al., Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, jama 316 (22) (2016) 2402--2410
work page 2016
-
[4]
D. S. W. Ting, C. Y.-L. Cheung, G. Lim, G. S. W. Tan, N. D. Quang, A. Gan, H. Hamzah, R. Garcia-Franco, I. Y. San Yeo, S. Y. Lee, et al., Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes, Jama 318 (22) (2017) 2211--2223
work page 2017
-
[5]
D. S. W. Ting, L. R. Pasquale, L. Peng, J. P. Campbell, A. Y. Lee, R. Raman, G. S. W. Tan, L. Schmetterer, P. A. Keane, T. Y. Wong, Artificial intelligence and deep learning in ophthalmology, British Journal of Ophthalmology 103 (2) (2019) 167--175
work page 2019
- [6]
-
[7]
J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Advances in neural information processing systems 33 (2020) 6840--6851
work page 2020
-
[8]
F. Shang, J. Fu, Y. Yang, H. Huang, J. Liu, L. Ma, Synfundus-1m: a high-quality million-scale synthetic fundus images dataset with fifteen types of annotation, arXiv preprint arXiv:2312.00377 (2023)
work page Pith review arXiv 2023
Show all 27 references
-
[9]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770--778
2016
-
[10]
M. Tan, Q. Le, Efficientnetv2: Smaller models and faster training, in: International conference on machine learning, PMLR, 2021, pp. 10096--10106
2021
-
[11]
S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, S. Xie, Convnext v2: Co-designing and scaling convnets with masked autoencoders, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 16133--16142
2023
-
[12]
Dosovitskiy, L
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020)
2010 arXiv
-
[13]
Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al., Swin transformer v2: Scaling up capacity and resolution, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12009--12019
2022
-
[14]
Y. Zhou, M. A. Chia, S. K. Wagner, M. S. Ayhan, D. J. Williamson, R. R. Struyven, T. Liu, M. Xu, M. G. Lozano, P. Woodward-Court, et al., A foundation model for generalizable disease detection from retinal images, Nature 622 (7981) (2023) 156--163
2023
-
[15]
Paszke, S
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems 32 (2019)
2019
-
[16]
Wightman, H
R. Wightman, H. Touvron, H. J \'e gou, Resnet strikes back: An improved training procedure in timm, arXiv preprint arXiv:2110.00476 (2021)
2021 arXiv
-
[17]
Buslaev, V
A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, A. A. Kalinin, Albumentations: fast and flexible image augmentations, Information 11 (2) (2020) 125
2020
-
[18]
Akiba, S
T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623--2631
2019
-
[19]
Loshchilov, F
I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017)
2017 arXiv
-
[20]
T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785--794
2016
-
[21]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, the Journal of machine Learning research 12 (2011) 2825--2830
2011
-
[22]
Dugas, Jared, Jorge, W
E. Dugas, Jared, Jorge, W. Cukierski, Diabetic retinopathy detection, https://kaggle.com/competitions/diabetic-retinopathy-detection, kaggle (2015)
2015
-
[23]
Dane, Aptos 2019 blindness detection, https://kaggle.com/competitions/aptos2019-blindness-detection, kaggle (2019)
Karthik, Maggie, S. Dane, Aptos 2019 blindness detection, https://kaggle.com/competitions/aptos2019-blindness-detection, kaggle (2019)
2019
-
[24]
Decenci \`e re, X
E. Decenci \`e re, X. Zhang, G. Cazuguel, B. Lay, B. Cochener, C. Trone, P. Gain, J.-R. Ord \'o \ n ez-Varela, P. Massin, A. Erginay, et al., Feedback on a publicly distributed image database: the messidor database, Image Analysis & Stereology (2014) 231--234
2014
-
[25]
de Vente, K
C. de Vente, K. A. Vermeer, N. Jaccard, H. Wang, H. Sun, F. Khader, D. Truhn, T. Aimyshev, Y. Zhanibekuly, T.-D. Le, A. Galdran, M. Ángel González Ballester, G. Carneiro, D. R. G, H. P. S, D. Puthussery, H. Liu, Z. Yang, S. Kondo, S. Kasai, E. Wang, A. Durvasula, J. Heras, M. ...
2023 arXiv
-
[26]
Pachade, P
S. Pachade, P. Porwal, D. Thulkar, M. Kokare, G. Deshmukh, V. Sahasrabuddhe, L. Giancardo, G. Quellec, F. M \'e riaudeau, Retinal fundus multi-disease image dataset (rfmid): A dataset for multi-disease detection research, Data 6 (2) (2021) 14
2021
-
[27]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 618--626
2017
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.