Pith. sign in

REVIEW 3 major objections 3 minor 22 references

Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Retinal foundation-model latent spaces enable controllable synthesis, but the resulting images fail independent real-image phenotype checks.

desk verdict A careful empirical evaluation of RepTok for retinal foundation models; the internal checks are circular by design and the external validation is genuinely useful but needs a reconstructed-real baseline and uncertainty quantification before the synthetic-to-real gap claim is solid. read the letter →

arxiv 2608.13455 v1 pith:D5HZLYTF submitted 2026-08-13 cs.CV

classification cs.CV
keywords colourfundusimagingdiffusionmodelfoundationsteerableimagegenerationretinalmodelssynthetic-to-realgapphenotypepreservationUKBiobank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the latent spaces of pretrained retinal foundation models can serve as a controllable substrate for generating fundus images that preserve demographic and clinical phenotypes. It reports that synthetic images and their representations do preserve conditioned phenotypes when judged by the foundation model that produced them, consistently beating a standard latent-diffusion baseline. But when the same images are judged by classifiers trained on real fundus images, these advantages largely disappear, exposing a synthetic-to-real representation gap. The paper identifies this gap as the main obstacle to using synthetic retinal images interchangeably with real ones in downstream clinical tasks.

What carries the argument

The Representation Tokenizer (RepTok) framework, a two-stage generative pipeline: first, a DiT-B decoder is trained to denoise VAE-compressed fundus images conditioned on the CLS token of a fine-tuned retinal foundation model under a flow-matching objective with cosine-similarity regularization; second, an MLP-Mixer is trained by flow matching to generate CLS-token embeddings from age, sex, BMI, hypertension, and three-year disease-onset metadata, with classifier-free guidance by randomly dropping conditioning inputs. This machinery lets a frozen foundation-model representation space act as the generative substrate, and it is the object whose in-model versus out-of-model phenotype preservation the paper measures.

What would settle it

Train the external classifiers on real images that have been passed through the same reconstruction pipeline (or an equivalent image-to-image transform), then re-run age, sex, and hypertension predictions on synthetic images; if the age $R^2$ gap shrinks to the same size as the drop on reconstructed real images, the synthetic-to-real gap is largely an artifact of classifier sensitivity to transformations rather than a genuine loss of phenotype information.

Watch

Extended reading notes

Core claim

Using the Representation Tokenizer framework with four retinal foundation models (RETFound, PRETI, FLAIR, URFound), the paper claims that generated representation tokens and decoded fundus images faithfully inherit conditioned demographic and clinical information when evaluated with the originating foundation model. On sex, age, hypertension, BMI, and three-year onset of myocardial infarction, stroke, and COPD, the synthetic images often match or exceed the phenotype signal carried by real-image representations. However, this advantage collapses under external evaluation with ResNet32 classifiers trained only on real images; only URFound retains an edge, and reconstructed real images already lose about 20 percentage points of $R^2$ in age prediction, indicating the external classifier is itself sensitive to image transformations. The claimed discovery is therefore two-sided: foundation-model latent spaces are a useful substrate for controllable retinal synthesis, yet synthetic representations are not yet aligned with real-image distributions.

Load-bearing premise

The external ResNet32 classifiers trained only on real images are treated as an unbiased sensor of phenotype preservation in synthetic images, even though the paper itself shows these classifiers lose about 20 percentage points of age explained variance on reconstructed real images.

Editorial extensions

If this is right

  • Foundation-model latent spaces can serve as a controllable substrate for phenotype-preserving fundus image synthesis when the originating model is the judge.
  • Synthetic representation tokens can carry near-ceiling conditioned phenotype information, but decoding them into images and re-encoding with independent classifiers reveals a substantial alignment loss.
  • Among the tested models, the multimodal URFound is the strongest across internal and external evaluations, while the CLIP-style FLAIR is weakest, suggesting pretraining strategy matters more than dataset scale for synthesis.
  • Conventional latent diffusion can match or beat RFM-based generation on external real-trained classifiers for several tasks, so the practical advantage of domain-specific latent spaces depends on the evaluation sensor.
  • Because reconstructed real images lose a large share of age explained variance under the external classifier, reported synthetic-to-real gaps should be read against a baseline of reconstruction-induced classifier sensitivity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the gap is real would be to train the external classifiers on real images passed through the same reconstruction transform; if age $R^2$ drops by the same amount on those images, part of the reported gap is an artifact of the evaluation sensor rather than of the synthesis.
  • The near-ceiling performance of generated tokens on conditioning attributes suggests the representation generator may place too much weight on metadata, and an independent test would be to generate under held-out combinations of attributes and check whether phenotype predictions reflect the manipulated variables rather than correlated confounds.
  • The results point toward domain-invariant representation learning or adversarial alignment between synthetic and real tokens as the natural next step, since the bottleneck is not in-model phenotype coding but cross-model transferability.
  • A practical consequence, if the gap persists, is that synthetic retinal images are better suited to training data augmentation for models that are later fine-tuned on real images than to direct clinical grading with off-the-shelf classifiers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper adapts the Representation Tokenizer (RepTok) framework to four retinal foundation models (RETFound, PRETI, FLAIR, URFound) and evaluates whether demographic and clinical attributes (age, sex, BMI, hypertension, and 3-year onset of MI, stroke, and COPD) are preserved when generating fundus images from foundation-model latent spaces. The authors train a generative decoder conditioned on fine-tuned foundation-model representations and a label-conditioned representation generator, then compare synthetic images against a latent diffusion baseline. Internally, they report near-ceiling phenotype prediction from synthetic tokens and synthetic images when evaluated with the originating foundation models. Externally, they evaluate with ResNet32 classifiers trained on real images and find the RepTok advantage largely disappears, which they interpret as a synthetic-to-real representation gap. The paper explicitly acknowledges the circularity of the internal evaluation and the sensitivity of the external classifiers to image transformations.

Significance. If the central claim were fully supported, the paper would make a useful contribution: it demonstrates that four diverse retinal foundation models can serve as generative latent spaces for controllable synthesis, it uses a large UK Biobank cohort with subject-level separation, and it includes a welcome external validation step rather than relying only on in-family evaluation. The authors are also transparent about the expected near-ceiling internal results and about the caution with which external classifier results must be read. However, the significance of the claimed synthetic-to-real representation gap depends on resolving the confounding introduced by the reconstructed-real baseline and on adding statistical support to the external comparisons; in its current form the evidence for the paper's headline conclusion is substantially weaker than the abstract suggests.

major comments (3)
  1. [§3.3, Table 3] The reconstructed-real row is the appropriate control for the claimed synthetic-to-real representation gap, and it undermines the conclusion as stated. Passing real images through the same VAE/decoder drops age R2 from 0.623 to 0.402, an absolute loss of 0.221 (about 35% relative). Since every synthetic image also passes through this decoder, the lower external scores of RepTok images relative to real images may reflect a common reconstruction-transformation gap rather than an RFM-specific representation shift. Against the reconstructed-real baseline, URFound is actually higher on age (0.541 vs 0.402) and hypertension (0.718 vs 0.698), and only sex is clearly lower (0.587 vs 0.695). The manuscript acknowledges this sensitivity in the discussion but does not quantify the comparison; the authors should report pairwise effect sizes or significance tests for synthetic versus reconstructed-real and revise the central claim accordingly.
  2. [§3.3, Table 3] All external validation numbers are reported as point estimates without confidence intervals or significance tests, even though the test set contains 11,780 unique subjects and bootstrap or analytic intervals are straightforward to compute. Claims such as "these gains largely disappeared," "latent diffusion generally showing stronger conditioning adherence," and "URFound slightly outperformed latent diffusion" are not statistically supported as reported. The authors should add confidence intervals and tests for the differences that carry the conclusion, especially for the URFound versus latent diffusion and URFound versus reconstructed-real comparisons.
  3. [§3.2, Figs. 3 and 4] The near-ceiling performance of synthetic tokens is expected by construction: the representation generator is conditioned on the same target labels that are later used for prediction, and the paper itself notes that this explains the strong performance. Because these internal results support the first half of the central claim (that generated representations and images "faithfully inherit phenotype information"), the evaluation needs a non-circular control. For example, the authors could compare against an unconditioned token generator with the same architecture, or evaluate attributes that were not used as conditioning inputs. Without such a control, the internal results demonstrate conditioning adherence but do not establish preservation of clinically meaningful structure beyond what was explicitly injected.
minor comments (3)
  1. [Table 1] The Latent Diffusion baseline row reports only gFID (24.13) and leaves rFID, PSNR, SSIM, and LPIPS blank, so the reconstruction comparison across methods is incomplete; please fill in these values or state explicitly why they are not reported.
  2. [Figs. 3 and 4] The bar charts show point estimates without error bars or confidence intervals; with 11,780 test subjects, confidence intervals should be feasible and would help the reader judge whether differences between models and token types are meaningful.
  3. [§2.1 and §2.2] There are small typographical issues: "1DMLP-Mixer" should read "1D MLP-Mixer," and "RETF ound" should read "RETFound."

Circularity Check

1 steps flagged · score 6.0 of 10

Internal phenotype-preservation evaluation reduces by construction: generated tokens are conditioned on target labels and then scored on those same labels with the same foundation model, while the external validation is independent.

  1. self definitional [Section 3.2, paragraph beginning 'We next evaluated whether synthetic tokens...' (yellow bars), immediately before the purple-bar paragraph.]
    "As shown, synthetic tokens achieved near-ceiling performance across all conditioned attributes, substantially outperforming representations extracted from real images. BMI remained the most challenging task but was still predicted considerably more accurately than from real-image representations. This strong performance is expected, as the representation generator is explicitly conditioned on the target demographic and clinical variables."

    In the second training stage (Sec. 2.1), the 1D MLP-Mixer representation generator is trained with classifier-free guidance to generate CLS-token embeddings from subject-specific demographic and clinical metadata, namely age, sex, BMI, hypertension, and 3-year MI/stroke/COPD onsets. At inference, the target labels are supplied as conditioning inputs. The near-ceiling results in Figs. 3-4 are obtained by taking these input-conditioned tokens, and images decoded from them, and predicting the same conditioning labels with the originating foundation model. The causal chain c -> z_synthetic -> c-hat makes c-hat approximate c by construction, so the evaluation cannot independently confirm phenotype preservation; it only confirms that the generator used its inputs.

full rationale

The paper contains one clear circular step. The representation generator is conditioned on demographic and clinical labels, and the internal evaluation then predicts those same labels from the generated tokens (and their decoded images) using the same foundation model. The near-ceiling performance is entailed by the construction, and the paper itself states it is expected. This circular check supports the abstract's first positive claim about faithful inheritance of phenotype information within the originating foundation models, so it is not merely decorative. However, the second, headline claim — the synthetic-to-real representation gap — is supported by the external ResNet32 classifiers trained on real images, which are statistically independent of the generation pipeline. That external measurement is not circular, and the paper's own reconstructed-real row shows that the external classifier is sensitive to reconstruction transformations; that is an important caveat about interpretation, but it is not a circularity. No load-bearing self-citation chain is present: reference [16] is a prior self-citation used as background motivation, and the RepTok framework [5] is external prior work. The score is 6 rather than higher because the central external gap claim retains independent content and is not forced by definition; the circularity affects the internal-preservation claim, which the paper nonetheless presents as a primary finding. A score lower than 6 would understate that the abstract's 'powerful substrate for controllable retinal synthesis' assertion leans on a self-consistent check that cannot be falsified by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No new free parameters or invented entities are introduced; the paper is an empirical comparison. The central claim rests on assumptions about the compression step, the evaluation probes, the external classifiers, and the data split, all of which are standard for this type of study but unverified by shipped artifacts.

assumptions (5)
  • domain assumption The pretrained VAE compresses colour fundus images without losing phenotype-relevant detail.
    Stage one decodes from VAE latents (Sec. 2.1); if compression discarded clinical detail, preservation results would be undermined.
  • domain assumption Linear-probe AUROC on real-image RFM representations is a valid proxy for clinically meaningful phenotype encoding.
    Used throughout Sec. 3.2 to define the real tokens baseline and to interpret synthetic-token performance.
  • domain assumption External ResNet32 classifiers trained on real images are unbiased sensors of phenotype preservation in synthetic images.
    Load-bearing for the synthetic-to-real gap in Sec. 3.3; the paper itself weakens this by noting classifier sensitivity to reconstruction artifacts.
  • domain assumption UK Biobank labels (hypertension, 3-year onsets) are accurate and the subject-unique train/test split prevents leakage.
    Both stages and the held-out evaluation rely on this (Sec. 2.3); label noise or subject leakage would bias internal and external performance.
  • domain assumption The RepTok framework is applicable to the four retinal foundation models with only the reported fine-tuning adjustments.
    The paper assumes that the attention-free DiT-B decoder and MLP-Mixer representation generator work for ViT and ResNet encoder variants without rederivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces." pith.science (2026). https://pith.science/paper/D5HZLYTF

@misc{pith2026260813455,
  author       = {Pith},
  title        = {Pith review of: Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5HZLYTF}},
  note         = {Machine review of arXiv:2608.13455}
}
read the original abstract

Medical foundation models learn latent representations of clinically meaningful phenotypes, yet their ability to support controllable image generation remains largely unexplored. We evaluate four retinal foundation models within the representation tokenizer framework and examine whether demographic and clinical information encoded in latent representations from foundation models is preserved during synthetic image generation. We show that generated representations and images faithfully inherit phenotype information when evaluated within their originating foundation models, consistently outperforming conventional latent diffusion on multiple downstream prediction tasks. However, these gains largely disappear when evaluated using classifiers trained on real images, revealing a previously uncharacterised synthetic-to-real representation gap. These findings demonstrate that foundation-model latent spaces provide a powerful substrate for controllable retinal synthesis while highlighting the need to better align synthetic representations with real-image distributions.

Figures

Figures reproduced from arXiv: 2608.13455 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. (a) A generative decoder is trained to re￾construct VAE-compressed image latents conditioned on fine-tuned retinal foundation model representations. (b) A representation generator trained on outputs (representa￾tions) from the frozen foundation model with demographic and clinical conditioning for controllable generation. 2.1 Representation Tokenizer To enable image synthesis, we e… view at source ↗
Figure 2
Figure 2. Real (first column) and synthetic fundus images generated by latent diffusion and RepTok conditioned on current hypertension and 3-year onset of chronic obstruc￾tive pulmonary disease (COPD), myocardial infarction (MI), and stroke. 3.1 Generation and Reconstruction Fidelity [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prediction performance on demographic and hypertension tasks using foun￾dation model representations from real images (blue), tokens from synthetic images (yellow), and synthetic images (purple). Overall, token from foundation model, tokens from synthetic images, and synthetic images all encode demographic information well, outperforming the baseline latent diffusion model. URFound PRETI RETFound FLAIR Latent Diffus… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Prediction performance for 3 year onset of systemic conditions using foundation model representations from real images (blue), tokens from synthetic images (yellow), and synthetic images (purple) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 6 canonical work pages

  1. [1]

    et al.: The UK Biobank resource with deep phe- notyping and genomic data

    Bycroft, C. et al.: The UK Biobank resource with deep phe- notyping and genomic data. Nature 2018 562:7726562(7726), 203–209 (10 2018). https://doi.org/10.1038/s41586-018-0579-z, https://www.nature.com/articles/s41586-018-0579-z

  2. [2]

    et al.: BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding

    Devlin, J. et al.: BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. NAACL HLT 2019 - 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies - Proceedings of the Conference1, 4171–4186 (10 2018), https://arxiv.org/pdf/1810.04805

  3. [3]

    et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, A. et al.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021 - 9th International Conference on Learning Rep- resentations (10 2020), https://arxiv.org/pdf/2010.11929

  4. [4]

    et al.: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

    Esser, P. et al.: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Proceedings of Machine Learning Research235, 12606–12633 (3 2024), https://arxiv.org/pdf/2403.03206

  5. [5]

    et al.: Adapting Self-Supervised Representations as a Latent Space for Efficient Generation (10 2025), https://arxiv.org/pdf/2510.14630

    Gui, M. et al.: Adapting Self-Supervised Representations as a Latent Space for Efficient Generation (10 2025), https://arxiv.org/pdf/2510.14630

  6. [7]

    et al.: Deep Residual Learning for Image Recognition

    He, K. et al.: Deep Residual Learning for Image Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2016-December, 770–778 (12 2015). https://doi.org/10.1109/CVPR.2016.90, https://arxiv.org/abs/1512.03385v1

  7. [8]

    Ho, J., Salimans, T.: Classifier-Free Diffusion Guidance (7 2022), https://arxiv.org/pdf/2207.12598

  8. [9]

    et al.: PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning

    Lee, Y. et al.: PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning. Lecture Notes in Computer Science 15960 LNCS, 523–533 (2026). https://doi.org/10.1007/978-3-032-04927-8\_50, https://link.springer.com/chapter/10.1007/978-3-032-04927-8_50 10 Z. Wakefield-Skorniewska et al

Show all 22 references
  1. [10]

    et al.: Flow Matching for Generative Modeling

    Lipman, Y. et al.: Flow Matching for Generative Modeling. 11th Inter- national Conference on Learning Representations, ICLR 2023 (10 2022), https://arxiv.org/pdf/2210.02747

  2. [11]

    https://doi.org/10.1038/s41586- 025-09079-8, https://www.nature.com/articles/s41586-025-09079-8

    Ma,D.A.etal.:AfullyopenAIfoundationmodelappliedtochestradiography.Na- ture 2025 643:8071643(8071), 488–498 (6 2025). https://doi.org/10.1038/s41586- 025-09079-8, https://www.nature.com/articles/s41586-025-09079-8

  3. [12]

    npj Imaging 2026 (5 2026)

    Mazher, M., Parker, G.J.M., Alexander, D.C.: Towards generalisable foundation models for brain MRI. npj Imaging 2026 (5 2026). https://doi.org/10.1038/S44303- 026-00176-5, https://www.nature.com/articles/s44303-026-00176-5

  4. [13]

    et al.: Learning Transferable Visual Models From Natural Language Supervision

    Radford, A. et al.: Learning Transferable Visual Models From Natural Language Supervision. Proceedings of Machine Learning Research139, 8748–8763 (2 2021), https://arxiv.org/pdf/2103.00020

  5. [14]

    et al.: High-Resolution Image Synthesis with Latent Dif- fusion Models

    Rombach, R. et al.: High-Resolution Image Synthesis with Latent Dif- fusion Models. In: Proceedings of the IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition. vol. 2022-June (2022). https://doi.org/10.1109/CVPR52688.2022.01042

  6. [15]

    et al.: A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision

    Silva-Rodríguez, J. et al.: A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision. Medical Image Analysis99, 103357 (1 2025). https://doi.org/10.1016/J.MEDIA.2024.103357, https://www.sciencedirect.com/science/article/pii/S136184...

  7. [16]

    In: Annual Confer- ence on Medical Image Understanding and Analysis

    Skórniewska, Z., Papież, B.W.: Exploring the effectiveness of deep features from domain-specific foundation models in retinal image synthesis. In: Annual Confer- ence on Medical Image Understanding and Analysis. pp. 293–305. Springer (2025)

  8. [17]

    In: ICLR 2021 - 9th International Conference on Learning Representations (2021)

    Song, J., Meng, C., Ermon, S.: DENOISING DIFFUSION IMPLICIT MODELS. In: ICLR 2021 - 9th International Conference on Learning Representations (2021)

  9. [18]

    et al.: Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

    Tiu, E. et al.: Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nature Biomedical Engineering 2022 6:126(12), 1399–1406 (9 2022). https://doi.org/10.1038/s41551-022-00936-9, https://www.nature.com/articles/s41551-022-00936-9

  10. [19]

    et al.: MLP-Mixer: An all-MLP Architecture for Vision

    Tolstikhin, I. et al.: MLP-Mixer: An all-MLP Architecture for Vision. Ad- vances in Neural Information Processing Systems29, 24261–24272 (5 2021), https://arxiv.org/pdf/2105.01601

  11. [20]

    et al.: BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning (4 2026), https://arxiv.org/pdf/2604.27277

    Wu, Y. et al.: BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning (4 2026), https://arxiv.org/pdf/2604.27277

  12. [21]

    et al.: UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling

    Yu, K. et al.: UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling. Lecture Notes in Computer Science (includ- ing subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinfor- matics)15012 LNCS, 753–762 (2024). https:/...

  13. [22]

    et al.: A foundation model for generalizable disease de- tection from retinal images

    Zhou, Y. et al.: A foundation model for generalizable disease de- tection from retinal images. Nature 2023 622:7981622(7981), 156–163 (9 2023). https://doi.org/10.1038/s41586-023-06555-x, https://www.nature.com/articles/s41586-023-06555-x

  14. [23]

    et al.: AutoMorph: Automated Retinal Vascular Morphology Quantification Via a Deep Learning Pipeline

    Zhou, Y. et al.: AutoMorph: Automated Retinal Vascular Morphology Quantification Via a Deep Learning Pipeline. Translational Vision Science & Technology11(7), 12–12 (7 2022). https://doi.org/10.1167/TVST.11.7.12, https://doi.org/10.1167/tvst.11.7.12

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.