REVIEW 4 major objections 5 minor 34 references
Exploring the Effectiveness of Deep Features from Domain-Specific Foundation Models in Retinal Image Synthesis
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Deep features from a retinal foundation model do not make synthetic colour fundus images more realistic or clinically accurate, while a simple edge-detection loss nearly matches a natural-image perceptual loss.
desk verdict Useful empirical comparison with a practical edge-loss takeaway, but the headline claim about domain-specific deep features is overgeneralized because the RETFound loss is confounded with architecture, layer count, and unreported loss weights. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the RETFound loss: the L1 distance in the 1024-dimensional embedding space of the last transformer block of RETFound, computed between real and synthetic images and added to a VQ-GAN training scheme alongside pixel and GAN losses. The comparison machinery includes perceptual loss, which sums L2 distances over five VGG-16 activation layers, and edge loss, which computes L2 distance between Meijering-filter vesselness responses on the green channel. Validation runs through FID, MMD, and MS-SSIM for fidelity, downstream Train-Synthetic/Test-Real prediction of cardiovascular risk factors, and permutation tests over 45 AutoMorph retinal morphology features.
What would settle it
Retrain the VQ-GAN with a RETFound-based loss that, like the perceptual loss, sums distances over several transformer layers instead of only the final layer. If FID, downstream prediction errors, and feature-overlap p-values then rise to match the perceptual-loss model, the claim that domain-specific deep features do not help would be contradicted.
Extended reading notes
Core claim
The central claim is that leveraging deep features of foundational models trained on domain data does not yield samples that are more realistic or that better preserve morphological and clinical information in colour fundus images. In head-to-head comparison, the VQ-GAN supervised by RETFound loss achieved FID 11.43 versus 2.45 for perceptual loss and 5.47 for edge loss, and in downstream clinical prediction tasks it performed worse than even the pixel-only baseline. Perceptually supervised models matched the largest share of real retinal features in permutation tests, while the RETFound-supervised model deviated most on fractal dimension and vascular complexity. The authors note that this is surprising given the established use of RETFound encoder features for cardiovascular risk prediction, and they list architectural differences, such as a vision transformer versus a CNN and reliance on a single activation layer, as possible explanations.
Load-bearing premise
The comparison assumes that the two losses differ mainly in whether the features come from a retinal foundation model or a natural-image network; in fact they also differ in number of network layers, so the disappointing results may come from the implementation rather than from domain-specific features.
Editorial extensions
If this is right
- For colour fundus synthesis, a domain-specific foundation-model perceptual loss built from a single final layer is not worth its compute relative to perceptual or edge losses.
- Simple edge-detection supervision is a practical, weight-free alternative that nearly matches expensive deep-feature losses for vascular sharpness.
- Downstream clinical prediction and morphological feature overlap should be part of evaluating synthetic medical images, since fidelity metrics such as MMD and MS-SSIM alone could not separate the models.
- No tested model matched real images in downstream prediction, so synthetic fundus data still cannot replace real data for these tasks.
- The result leaves open whether transformer-based losses or multi-layer perceptual losses could improve synthesis, because the RETFound loss used only one activation layer.
Reading between the lines
- Editorial inference: the paper does not isolate whether the failure comes from domain-specific features, from the vision-transformer architecture, or from using only one activation layer; a fair test would need a multi-layer RETFound loss or a CNN retinal backbone.
- Editorial inference: if the single-layer implementation drives the gap, the broader conclusion that domain-specific features do not help would be too hasty and should be tested directly.
- Editorial inference: the strong edge-loss result hints that low-level vessel geometry is the bottleneck for clinical utility, more than high-level semantic features; combining edge loss with perceptual loss could test this.
- Editorial inference: the validation design could serve as a template for other medical modalities, reporting feature-overlap and downstream-task performance rather than relying on FID alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether a perceptual-style loss built on deep features of RETFound, a self-supervised foundation model trained on 1.6 million colour fundus images, improves VQ-GAN-based synthesis of retinal images compared with a VGG-16 perceptual loss, an edge-detection (Meijering filter) loss, and a pixel-only baseline. All models are trained on 90,344 UK Biobank colour fundus images at 256x256 resolution and evaluated on FID/MMD/MS-SSIM, on Train-Synthetic/Test-Real prediction of age, sex, BMI, blood pressure, HbA1c and glucose using AutoMorph-extracted features, and on distributional matching of 45 retinal morphology features via permutation tests. The authors conclude in Section 5 that deep features from a domain-specific foundation model do not yield more realistic or morphologically accurate synthetic fundus images, and that a conventional edge-based loss is a cheap and competitive alternative.
Significance. If established, the negative result is a useful contribution to medical image synthesis: it challenges the implicit assumption that domain-specific foundation-model features are the right backbone for perceptual-style losses in this setting, and it identifies edge-based supervision as a practical, low-cost alternative. The study has genuine methodological strengths: the evaluation pipeline is unusually comprehensive for the area, spanning fidelity, downstream clinical prediction, and direct morphological comparison; the evaluation metrics and feature extractors (AutoMorph, RETFound, InceptionV3) are external pretrained models independent of the training losses, so circularity is minimal; and the central claim is a clearly falsifiable negative result rather than another positive demonstration on a single metric. The main weakness is that the headline claim is stated more broadly than the experiments can support: the RETFound loss and the perceptual loss differ in architecture, layer count, distance norm, resolution, and unreported loss weights, and the paper's own Discussion concedes that these factors are plausible explanations for the observed gap.
major comments (4)
- [§2.1, §3.3, §3.5, §5] The central negative claim that domain-specific deep features are ineffective rests on comparing VQ-GAN (RETFound) with VQ-GAN (Perceptual), but the two losses differ along several axes at once. The RETFound loss (Section 3.3) is an L1 distance on a single 1024-dimensional final-layer embedding of a ViT-Large, computed after resizing inputs to 224x224, whereas the perceptual loss (Section 3.5) is a sum of L2 distances over five VGG-16 activation layers (4th, 9th, 16th, 23rd, 30th) at the native 256x256 resolution. Feature source, architecture (ViT vs CNN), layer count (1 vs 5), distance norm (L1 vs L2), and resolution are therefore all varied simultaneously, and the weighting of the auxiliary loss terms relative to the pixel and GAN losses is never reported (Section 2.1 lists three gradient sources without coefficients). The Discussion (Section 5) itself names the architecture difference and the layer-count difference as possible reasons for the RETFound loss underperforming, which concedes that the experiment does not isolate the effect of feature source. As stated, the headline conclusion overgeneralizes from a single confounded configuration; the authors should either add ablations that match layer count and architecture (for example, a multi-layer RETFound loss or a single-layer VGG loss) or narrow the claim to the specific loss configuration tested.
- [§4.2, Table 2] The sentence in Section 4.2 that 'none of the synthetic models achieved error rates as low as those based on real data' is contradicted by the paper's own Table 2: for age, VQ-GAN (Perceptual) reports MAE 7.60 (0.27), which is lower than the real-data MAE of 7.72 (0.13); for sex, VQ-GAN (RETFound) reports F1 0.53 (0.01), higher than the real-data 0.52 (0.01). The blanket statement is therefore factually incorrect as written and should be replaced with a per-variable comparison or a qualified statement such as 'in most prediction tasks.'
- [§4.1, Tables 1–2, §4.3] FID and MMD are reported in Table 1 as single point values with no error bars, and the number of training runs or evaluation seeds is not stated anywhere. For a paper whose main contribution is a negative result, the absence of uncertainty quantification is load-bearing: the FID spread across models (2.45, 5.47, 11.43, 19.95) and the morphological 'matches' counted in Section 4.3 (3, 3, 4, and 5 of 45 features) are small-number comparisons, and without confidence intervals, multiple seeds, or a statistical test the reader cannot judge whether the ordering of the models is stable. This is especially important for the claim that the RETFound loss does not improve over the pixel-only baseline, since its FID of 11.43 is closer to real data than the pixel baseline's 19.95 but this difference may be within run-to-run noise.
- [§5, Table 1] The Discussion states that 'the RETFound loss consistently underperformed,' but Table 1 shows that VQ-GAN (RETFound) achieves FID 11.43 versus 19.95 for VQ-GAN (Pixel), i.e., the RETFound loss substantially improves fidelity relative to the pixel-only baseline on this metric. The abstract's claim that domain-specific deep features 'do not improve autoencoder image generation' is likewise in tension with the paper's own evidence: the RETFound loss improves FID over the baseline while worsening downstream-task performance and morphological matching. The paper's conclusions should be reframed as a mixed, metric-dependent result, with the specific negative finding being that RETFound features underperform VGG perceptual features and edge-based supervision on most downstream and morphological measures.
minor comments (5)
- [Abstract] Several LaTeX or line-break artifacts remain in the text ('Fr\'echet', 'in-vestigated', 'autoen-coder'); these should be cleaned up before final submission.
- [Table 2] Table 2 uses the label 'VQGAN (RETFound)' while the text and Table 1 use 'VQ-GAN (RETFound)'; the notation should be unified.
- [§3.1, §4.1] Section 3.1 does not state how the 90,344 UK Biobank images were split into training, validation, and test sets, and Section 4.1 does not say whether the 'Real' row in Table 1 is computed on the same 20,000-image subset used for the synthetic comparisons; both details should be reported.
- [Fig. 3] The Figure 3 caption claims on the basis of a single sample that VQ-GAN (RETFound) 'struggles to replicate the optic cup'; this qualitative observation should either be tied to the quantitative optic-disc/cup features in Figure 4 or softened.
- [§5, ref. [33]] The citation of PeCo [33] to argue that transformer-based perceptual losses outperform pixel-based losses is not directly applicable, since PeCo is a pre-training objective rather than a frozen-feature reconstruction loss; the relevance to the present setup should be explained or the sentence revised.
Circularity Check
No circularity: the comparison is empirical and evaluation metrics are independent of training losses; confounds are acknowledged, not hidden.
full rationale
This paper makes an empirical comparison among VQ-GAN training losses: RETFound loss, perceptual loss, edge loss, and pixel-only baseline. There is no derivation chain in which a predicted quantity is defined in terms of the fitted quantity. RETFound, VGG-16, and AutoMorph are external pretrained models used as fixed feature extractors or evaluation tools; their weights are not trained or fitted on the UK Biobank data in this study. The evaluation metrics (FID, MMD, MS-SSIM, AutoMorph-based downstream prediction errors, and permutation tests on extracted retinal features) are computed from the generated samples and are not used as training objectives, so there is no fitted-input-called-prediction pattern. The central negative claim that domain-specific deep features do not improve synthesis is an inductive generalization from a comparative experiment, not a tautology. The skeptic concern that the RETFound loss and perceptual loss differ along several axes at once (architecture, number of layers, distance norm, loss weight reporting) is a confounding/overgeneralization issue about external validity, not circularity. The paper itself flags these confounds in Section 5, noting that the underperformance 'may also stem from differences in model architecture' and that 'another possible reason for the relative drop in RETFound loss performance compared to perceptual loss is the difference in the number of activation layers used.' That acknowledgment further shows the result is not presented as forced by construction. The only self-citations (references 9, 21, 22) are contextual prior work by overlapping authors and are not load-bearing for the central claim. No equation in the paper reduces to its own input, and no uniqueness theorem or ansatz is imported via self-citation. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Auxiliary loss weighting coefficients =
not reported
- Layer set for RETFound loss =
1 (final transformer block)
- Resizing resolution for RETFound =
224x224
assumptions (4)
- domain assumption RETFound final-layer embeddings capture clinically relevant retinal morphology
- domain assumption AutoMorph features are a valid proxy for morphological and clinical accuracy
- domain assumption Train-Synthetic, Test-Real with AutoMorph features measures clinical usefulness
- domain assumption UK Biobank 90,344 images are representative for the comparison
Cite this review
Pith. "Pith review of Exploring the Effectiveness of Deep Features from Domain-Specific Foundation Models in Retinal Image Synthesis." pith.science (2026). https://pith.science/paper/SF4SKGEK
@misc{pith2026250611753,
author = {Pith},
title = {Pith review of: Exploring the Effectiveness of Deep Features from Domain-Specific Foundation Models in Retinal Image Synthesis},
year = {2026},
howpublished = {\url{https://pith.science/paper/SF4SKGEK}},
note = {Machine review of arXiv:2506.11753}
}
read the original abstract
The adoption of neural network models in medical imaging has been constrained by strict privacy regulations, limited data availability, high acquisition costs, and demographic biases. Deep generative models offer a promising solution by generating synthetic data that bypasses privacy concerns and addresses fairness by producing samples for under-represented groups. However, unlike natural images, medical imaging requires validation not only for fidelity (e.g., Fr\'echet Inception Score) but also for morphological and clinical accuracy. This is particularly true for colour fundus retinal imaging, which requires precise replication of the retinal vascular network, including vessel topology, continuity, and thickness. In this study, we in-vestigated whether a distance-based loss function based on deep activation layers of a large foundational model trained on large corpus of domain data, colour fundus imaging, offers advantages over a perceptual loss and edge-detection based loss functions. Our extensive validation pipeline, based on both domain-free and domain specific tasks, suggests that domain-specific deep features do not improve autoen-coder image generation. Conversely, our findings highlight the effectiveness of con-ventional edge detection filters in improving the sharpness of vascular structures in synthetic samples.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
R. Poplin et al., ‘Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning’, Nat Biomed Eng, vol. 2, no. 3, pp. 158–164, Feb. 2018, doi: 10.1038/s41551-018-0195-0
-
[2]
C. Y. Cheung et al., ‘A deep-learning system for the assessment of cardiovas- cular disease risk via the measurement of retinal-vessel calibre’, Nat Biomed Eng, vol. 5, no. 6, 2021, doi: 10.1038/s41551-020-00626-4
-
[3]
Y. Zhou et al., ‘A foundation model for generalizable disease detection from retinal images’, Nature, vol. 622, no. 7981, pp. 156–163, Sep. 2023, doi: 10.1038/s41586-023-06555-x
-
[4]
C. Y. Cheung et al., ‘A deep learning model for detection of Alzheimer’s dis- ease based on retinal photographs: a retrospective, multicentre case-control study’, Lancet Digit Health, vol. 4, no. 11, 2022, doi: 10.1016/S2589- 7500(22)00169-8
doi:10.1016/s2589- 2022
-
[5]
G. An et al., ‘Glaucoma Diagnosis with Machine Learning Based on Optical Coherence Tomography and Color Fundus Images’, J Healthc Eng, vol. 2019, 2019, doi: 10.1155/2019/4061313
-
[6]
Y. Zhou et al., ‘AutoMorph: Automated Retinal Vascular Morphology Quan- tification Via a Deep Learning Pipeline’, Translational Vision Science & Technology, vol. 11, no. 7, p. 12, Jul. 2022, doi: 10.1167/tvst.11.7.12
- [7]
-
[8]
A. Hoover, ‘Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response’, IEEE Trans Med Imaging, vol. 19, no. 3, 2000, doi: 10.1109/42.845178
Show all 34 references
-
[9]
Alloula, R
A. Alloula, R. Mustafa, D. R. McGowan, and B. W. Papież, ‘On Biases in a UK Biobank-based Retinal Image Classification Model’, 2024. [Online]. Available: https://arxiv.org/abs/2408.02676
2024 arXiv
-
[10]
Esser, R
P. Esser, R. Rombach, and B. Ommer, ‘Taming Transformers for High-Reso- lution Image Synthesis’, 2021. [Online]. Available: https://arxiv.org/abs/2012.09841
2021 arXiv
-
[11]
Rombach, A
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, ‘High-Reso- lution Image Synthesis with Latent Diffusion Models’, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition, 2022. doi: 10.1109/CVPR52688.2022.01042. 13
2022
-
[12]
J. Ho, A. Jain, and P. Abbeel, ‘Denoising diffusion probabilistic models’, in Advances in Neural Information Processing Systems, 2020
2020
-
[13]
J. Song, C. Meng, and S. Ermon, ‘DENOISING DIFFUSION IMPLICIT MODELS’, in ICLR 2021 - 9th International Conference on Learning Repre- sentations, 2021
2021
- [14]
- [15]
-
[16]
Litrico, F
M. Litrico, F. Guarnera, M. V. Giuffrida, D. Ravì, and S. Battiato, ‘TADM: Temporally-Aware Diffusion Model for Neurodegenerative Progression on Brain MRI’, in proceedings of Medical Image Computing and Computer As- sisted Intervention – MICCAI 2024, Springer Nature Switzerlan...
2024
-
[17]
N. Konz, Y. Chen, H. Dong, and M. A. Mazurowski, ‘Anatomically-Control- lable Medical Image Generation with Segmentation-Guided Diffusion Mod- els’, in proceedings of Medical Image Computing and Computer Assisted In- tervention – MICCAI 2024, Springer Nature Switzerland, Oct. 2024
2024
-
[18]
Doerrich, F
S. Doerrich, F. Di Salvo, and C. Ledig, ‘Self-supervised Vision Transformer are Scalable Generative Models for Domain Generalization’, in proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, Springer Nature Switzerland, Oct. 2024
2024
-
[19]
Sturm, L
M. Sturm, L. Cerrone, and F. A. Hamprecht, ‘SynCellFactory: Generative Data Augmentation for Cell Tracking’, in proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, Springer Nature Switzerland, Oct. 2024
2024
- [20]
-
[21]
Bradbury, K
R. Bradbury, K. A. Vallis, and B. W. Papiez, ‘Paired Diffusion: Generation of related, synthetic PET-CT-Segmentation scans using Linked Denoising Diffu- sion Probabilistic Models’, Proceedings - International Symposium on Bio- medical Imaging, Mar. 2024, doi: 10.1109/ISBI56570...
2024
-
[22]
Zhang, F
L. Zhang, F. Wu, K. Bronik, and B. W. Papiez, ‘DiffuSeg: Domain-driven Diffusion for Medical Image Segmentation’, IEEE J Biomed Health Inform, 2025, doi: 10.1109/JBHI.2025.3526806
2025
-
[23]
Hou et al., ‘FundusGAN: A Hierarchical Feature-Aware Generative Framework for High-Fidelity Fundus Image Generation’, Mar
Q. Hou et al., ‘FundusGAN: A Hierarchical Feature-Aware Generative Framework for High-Fidelity Fundus Image Generation’, Mar. 2025, Ac- cessed: Apr. 05, 2025. [Online]. Available: https://arxiv.org/abs/2503.17831v1
2025 arXiv
-
[24]
S. Ahn, S. J. Song, and J. Shin, ‘FundusGAN: Fundus image synthesis based on semi-supervised learning’, Biomed Signal Process Control, vol. 86, p. 105289, Sep. 2023, doi: 10.1016/J.BSPC.2023.105289
2023
-
[25]
Zhang, P
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, ‘The Unreason- able Effectiveness of Deep Features as a Perceptual Metric’, in 2018 14 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 586–595. doi: 10.1109/CVPR.2018.00068
2018
-
[26]
K. He, X. Zhang, S. Ren, and J. Sun, ‘Deep Residual Learning for Image Recognition’, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-December, pp. 770–778, Dec. 2015, doi: 10.1109/CVPR.2016.90
2016 doi
-
[27]
van den Oord, O
A. van den Oord, O. Vinyals, and K. Kavukcuoglu, ‘Neural Discrete Repre- sentation Learning’, 2018. [Online]. Available: https://arxiv.org/abs/1711.00937
2018 arXiv
-
[28]
Dosovitskiy et al., ‘An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale’, 2021
A. Dosovitskiy et al., ‘An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale’, 2021. [Online]. Available: https://arxiv.org/abs/2010.11929
2021 arXiv
-
[29]
Meijering, M
E. Meijering, M. Jacob, J. C. F. Sarria, P. Steiner, H. Hirling, and M. Unser, ‘Design and Validation of a Tool for Neurite Tracing and Analysis in Fluores- cence Microscopy Images’, Cytometry Part A, vol. 58, no. 2, 2004, doi: 10.1002/cyto.a.20022
2004 doi
-
[30]
Simonyan and A
K. Simonyan and A. Zisserman, ‘Very Deep Convolutional Networks for Large-Scale Image Recognition’, 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, Sep. 2014, Ac- cessed: Apr. 06, 2025. [Online]. Available: https://arxiv.org/...
2015 arXiv
-
[31]
Szegedy, V
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ‘Rethinking the Inception Architecture for Computer Vision’, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition ,
-
[32]
Jordon et al., ‘Synthetic Data -- what, why and how?’, May 2022
J. Jordon et al., ‘Synthetic Data -- what, why and how?’, May 2022
2022
-
[33]
Dong et al., ‘PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers’, Proceedings of the AAAI Conference on Artificial Intelligence, vol
X. Dong et al., ‘PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers’, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, pp. 552–560, Jun. 2023, doi: 10.1609/AAAI.V37I1.25130
2023 doi
-
[2016]
doi: 10.1109/CVPR.2016.308
2016 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.