{"id":"e95839a6-a4d5-4f51-8758-0e7ab941874e","arxiv_id":"2608.13455","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Retinal foundation model latents support controllable image synthesis that preserves clinical phenotypes under self-evaluation, but external classifiers show a synthetic-to-real representation gap.","lead":"This paper tests whether four retinal image AI models can generate synthetic eye images that keep key patient details like age, sex, and disease history. It finds the generated images pass checks inside the same AI system, but the gains shrink when checked by a separate system trained on real photos, exposing a synthetic to real gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"External real-trained classifiers are a confounded sensor: reconstructed real images already lose ~35% of age R2 (0.623→0.402 in Table 3), so the synthetic-to-real gap may reflect the decoder/VAE transformation rather than an RFM-specific representation shift.","rationale":"The reader's weakest assumption correctly identified the external ResNet32 classifiers as an unreliable sensor because reconstructed real images already lose substantial age explained variance. My stress-test confirms this is the most load-bearing point: the paper's headline claim about a synthetic-to-real gap depends entirely on this external evaluation. The reconstructed-real row is present in Table 3, but it is used only as a cautionary note, not as the proper baseline for interpreting synthetic results. A concrete comparison against reconstructed-real, with uncertainty quantification, would settle whether the gap is specific to synthetic generation or an artifact of the decoder transformation. Since the reader's CONDITIONAL verdict already hinges on this limitation, my read does not change the verdict; it sharpens the required condition. The paper's internal near-ceiling results are appropriately flagged as expected given conditioning, and the external evaluation is the only non-circular evidence for the gap, so the issue is not an internal inconsistency but an under-controlled comparison.","tokens_in":7025,"tokens_out":5186,"duration_ms":62109,"concrete_test":"Re-run Table 3 with an additional baseline: train the external ResNet32 classifiers on reconstructed-real images (real images passed through the VAE and generative decoder with RFM conditioning, as in the Table 3 row 2 protocol) and report bootstrap confidence intervals for every entry. If RepTok synthetic images do not differ significantly from reconstructed-real images on age, sex, and hypertension, the 'synthetic-to-real representation gap' should be reframed as a decoder-transformation gap common to all generated images, not an RFM-specific limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The second half of the central claim — that RFM synthesis gains 'largely disappear' under classifiers trained on real images — rests on Table 3, where ResNet32 classifiers are treated as an unbiased sensor of phenotype preservation. The reconstructed-real row undermines this. Real images passed through the same VAE/decoder lose 0.221 of age R2 (0.623 to 0.402), about 35% relative. Since every synthetic image also passes through this decoder, the low external scores on RepTok images may reflect a common reconstruction-transformation gap rather than a property of RFM latent spaces. This is not merely hypothetical: URFound's synthetic age R2 (0.541) exceeds reconstructed-real R2 (0.402), and its hypertension accuracy (0.718) exceeds reconstructed-real (0.698); only sex is clearly lower (0.587 vs 0.695). Thus the appropriate control for a 'synthetic-to-real' claim is not real images but reconstructed-real images, and against that baseline the gap shrinks or reverses for the best model. The paper acknowledges this sensitivity in Sec. 3.3 but still frames the result as a synthetic-to-real representation gap without quantifying whether the difference from reconstructed-real is significant. Without error bars or significance tests, the observed external gap cannot be attributed specifically to synthetic generation rather than to the decoder's lossy reconstruction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the Representation Tokenizer (RepTok) framework to four retinal foundation models (RETFound, PRETI, FLAIR, URFound) and evaluates whether demographic and clinical attributes (age, sex, BMI, hypertension, and 3-year onset of MI, stroke, and COPD) are preserved when generating fundus images from foundation-model latent spaces. The authors train a generative decoder conditioned on fine-tuned foundation-model representations and a label-conditioned representation generator, then compare synthetic images against a latent diffusion baseline. Internally, they report near-ceiling phenotype prediction from synthetic tokens and synthetic images when evaluated with the originating foundation models. Externally, they evaluate with ResNet32 classifiers trained on real images and find the RepTok advantage largely disappears, which they interpret as a synthetic-to-real representation gap. The paper explicitly acknowledges the circularity of the internal evaluation and the sensitivity of the external classifiers to image transformations.","tokens_in":7260,"tokens_out":3924,"duration_ms":44943,"significance":"If the central claim were fully supported, the paper would make a useful contribution: it demonstrates that four diverse retinal foundation models can serve as generative latent spaces for controllable synthesis, it uses a large UK Biobank cohort with subject-level separation, and it includes a welcome external validation step rather than relying only on in-family evaluation. The authors are also transparent about the expected near-ceiling internal results and about the caution with which external classifier results must be read. However, the significance of the claimed synthetic-to-real representation gap depends on resolving the confounding introduced by the reconstructed-real baseline and on adding statistical support to the external comparisons; in its current form the evidence for the paper's headline conclusion is substantially weaker than the abstract suggests.","major_comments":[{"comment":"The reconstructed-real row is the appropriate control for the claimed synthetic-to-real representation gap, and it undermines the conclusion as stated. Passing real images through the same VAE/decoder drops age R2 from 0.623 to 0.402, an absolute loss of 0.221 (about 35% relative). Since every synthetic image also passes through this decoder, the lower external scores of RepTok images relative to real images may reflect a common reconstruction-transformation gap rather than an RFM-specific representation shift. Against the reconstructed-real baseline, URFound is actually higher on age (0.541 vs 0.402) and hypertension (0.718 vs 0.698), and only sex is clearly lower (0.587 vs 0.695). The manuscript acknowledges this sensitivity in the discussion but does not quantify the comparison; the authors should report pairwise effect sizes or significance tests for synthetic versus reconstructed-real and revise the central claim accordingly.","section":"§3.3, Table 3"},{"comment":"All external validation numbers are reported as point estimates without confidence intervals or significance tests, even though the test set contains 11,780 unique subjects and bootstrap or analytic intervals are straightforward to compute. Claims such as \"these gains largely disappeared,\" \"latent diffusion generally showing stronger conditioning adherence,\" and \"URFound slightly outperformed latent diffusion\" are not statistically supported as reported. The authors should add confidence intervals and tests for the differences that carry the conclusion, especially for the URFound versus latent diffusion and URFound versus reconstructed-real comparisons.","section":"§3.3, Table 3"},{"comment":"The near-ceiling performance of synthetic tokens is expected by construction: the representation generator is conditioned on the same target labels that are later used for prediction, and the paper itself notes that this explains the strong performance. Because these internal results support the first half of the central claim (that generated representations and images \"faithfully inherit phenotype information\"), the evaluation needs a non-circular control. For example, the authors could compare against an unconditioned token generator with the same architecture, or evaluate attributes that were not used as conditioning inputs. Without such a control, the internal results demonstrate conditioning adherence but do not establish preservation of clinically meaningful structure beyond what was explicitly injected.","section":"§3.2, Figs. 3 and 4"}],"minor_comments":[{"comment":"The Latent Diffusion baseline row reports only gFID (24.13) and leaves rFID, PSNR, SSIM, and LPIPS blank, so the reconstruction comparison across methods is incomplete; please fill in these values or state explicitly why they are not reported.","section":"Table 1"},{"comment":"The bar charts show point estimates without error bars or confidence intervals; with 11,780 test subjects, confidence intervals should be feasible and would help the reader judge whether differences between models and token types are meaningful.","section":"Figs. 3 and 4"},{"comment":"There are small typographical issues: \"1DMLP-Mixer\" should read \"1D MLP-Mixer,\" and \"RETF ound\" should read \"RETFound.\"","section":"§2.1 and §2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, but the central claim about a synthetic-to-real representation gap currently rests on external comparisons that are confounded by the decoder transformation and are reported without uncertainty quantification. This is fixable within the scope of the manuscript through additional analysis against the reconstructed-real baseline and statistical testing. The fit with the journal is otherwise appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper applies the RepTok framework to four retinal foundation models and reports a result worth knowing: generated images and tokens look phenotype-rich when scored with the originating foundation model, but most of that advantage vanishes under classifiers trained on real images. That external validation is the real contribution, and it is a good one. The authors also deserve credit for flagging the circularity in their internal near-ceiling results, for including real, reconstructed-real, and latent-diffusion baselines, and for using a held-out set of 11,780 unique subjects.\n\nNow the soft spots, and they are in proportion to the claim. The central synthetic-to-real gap rests almost entirely on Table 3, which has no confidence intervals and no significance tests. More importantly, the stress-test concern is on target: the reconstructed-real row already loses about 35% of age R2 (0.623 to 0.402), and URFound's synthetic age R2 of 0.541 beats that reconstructed baseline, as does its hypertension accuracy. So the 'gap' may be substantially a decoder/VAE reconstruction effect rather than a property of RFM latent spaces. The paper acknowledges the sensitivity in Sec. 3.3 but still frames the result as a synthetic-to-real representation gap without quantifying whether RFM synthetic images actually differ from reconstructed-real images. That is the missing control, and it is a load-bearing one.\n\nMinor issues: no code or hyperparameters, small incidence cohorts (n<35) for phenotype-specific generation, and the external ResNet evaluation is restricted to age, sex, and hypertension. These are not fatal, but they limit reproducibility.\n\nOn balance, the paper is honest and well structured. The internal evaluation is circular by design and the authors say so; the external evaluation is the right idea but under-analyzed. I think the qualitative conclusion—\"internal RFM evaluation overstates transfer to real-trained models\"—is likely robust, but the precise magnitude and attribution are not yet established.\n\nThis deserves a serious referee, not a desk reject. It needs a revision that adds error bars, significance tests, and an explicit reconstructed-real baseline, and ideally code. I would bring it to a reading group focused on medical image synthesis or representation learning evaluation, and I would cite it cautiously as evidence for the evaluation gap.","headline":"A careful empirical evaluation of RepTok for retinal foundation models; the internal checks are circular by design and the external validation is genuinely useful but needs a reconstructed-real baseline and uncertainty quantification before the synthetic-to-real gap claim is solid.","tokens_in":7788,"tokens_out":1716,"would_cite":true,"duration_ms":21677,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retinal foundation-model latent spaces enable controllable synthesis, but the resulting images fail independent real-image phenotype checks.","keywords":["colour fundus imaging","diffusion model","foundation model","steerable image generation","retinal foundation models","synthetic-to-real gap","phenotype preservation","UK Biobank"],"falsifier":"Train the external classifiers on real images that have been passed through the same reconstruction pipeline (or an equivalent image-to-image transform), then re-run age, sex, and hypertension predictions on synthetic images; if the age $R^2$ gap shrinks to the same size as the drop on reconstructed real images, the synthetic-to-real gap is largely an artifact of classifier sensitivity to transformations rather than a genuine loss of phenotype information.","tokens_in":6821,"feed_emoji":"👁️","tokens_out":5456,"duration_ms":46442,"temperature":0.7,"pith_summary":"This paper asks whether the latent spaces of pretrained retinal foundation models can serve as a controllable substrate for generating fundus images that preserve demographic and clinical phenotypes. It reports that synthetic images and their representations do preserve conditioned phenotypes when judged by the foundation model that produced them, consistently beating a standard latent-diffusion baseline. But when the same images are judged by classifiers trained on real fundus images, these advantages largely disappear, exposing a synthetic-to-real representation gap. The paper identifies this gap as the main obstacle to using synthetic retinal images interchangeably with real ones in downstream clinical tasks.","feed_headline":"Retinal synthesis passes its own model, fails real-image checks","feed_subtitle":"Generated fundus images keep clinical phenotypes in foundation-model latent spaces, but independent classifiers expose a synthetic-to-real…","key_machinery":"The Representation Tokenizer (RepTok) framework, a two-stage generative pipeline: first, a DiT-B decoder is trained to denoise VAE-compressed fundus images conditioned on the CLS token of a fine-tuned retinal foundation model under a flow-matching objective with cosine-similarity regularization; second, an MLP-Mixer is trained by flow matching to generate CLS-token embeddings from age, sex, BMI, hypertension, and three-year disease-onset metadata, with classifier-free guidance by randomly dropping conditioning inputs. This machinery lets a frozen foundation-model representation space act as the generative substrate, and it is the object whose in-model versus out-of-model phenotype preservation the paper measures.","core_discovery":"Using the Representation Tokenizer framework with four retinal foundation models (RETFound, PRETI, FLAIR, URFound), the paper claims that generated representation tokens and decoded fundus images faithfully inherit conditioned demographic and clinical information when evaluated with the originating foundation model. On sex, age, hypertension, BMI, and three-year onset of myocardial infarction, stroke, and COPD, the synthetic images often match or exceed the phenotype signal carried by real-image representations. However, this advantage collapses under external evaluation with ResNet32 classifiers trained only on real images; only URFound retains an edge, and reconstructed real images already lose about 20 percentage points of $R^2$ in age prediction, indicating the external classifier is itself sensitive to image transformations. The claimed discovery is therefore two-sided: foundation-model latent spaces are a useful substrate for controllable retinal synthesis, yet synthetic representations are not yet aligned with real-image distributions.","pith_inferences":["A direct test of whether the gap is real would be to train the external classifiers on real images passed through the same reconstruction transform; if age $R^2$ drops by the same amount on those images, part of the reported gap is an artifact of the evaluation sensor rather than of the synthesis.","The near-ceiling performance of generated tokens on conditioning attributes suggests the representation generator may place too much weight on metadata, and an independent test would be to generate under held-out combinations of attributes and check whether phenotype predictions reflect the manipulated variables rather than correlated confounds.","The results point toward domain-invariant representation learning or adversarial alignment between synthetic and real tokens as the natural next step, since the bottleneck is not in-model phenotype coding but cross-model transferability.","A practical consequence, if the gap persists, is that synthetic retinal images are better suited to training data augmentation for models that are later fine-tuned on real images than to direct clinical grading with off-the-shelf classifiers."],"forward_implications":["Foundation-model latent spaces can serve as a controllable substrate for phenotype-preserving fundus image synthesis when the originating model is the judge.","Synthetic representation tokens can carry near-ceiling conditioned phenotype information, but decoding them into images and re-encoding with independent classifiers reveals a substantial alignment loss.","Among the tested models, the multimodal URFound is the strongest across internal and external evaluations, while the CLIP-style FLAIR is weakest, suggesting pretraining strategy matters more than dataset scale for synthesis.","Conventional latent diffusion can match or beat RFM-based generation on external real-trained classifiers for several tasks, so the practical advantage of domain-specific latent spaces depends on the evaluation sensor.","Because reconstructed real images lose a large share of age explained variance under the external classifier, reported synthetic-to-real gaps should be read against a baseline of reconstruction-induced classifier sensitivity."],"supporting_citations":[{"why":"Introduces the Representation Tokenizer framework that the paper adapts for retinal foundation models.","marker":"[5]"},{"why":"Provides RETFound, a vision-only retinal foundation model trained on hospital-sourced fundus images.","marker":"[22]"},{"why":"Provides PRETI, a patient-aware retinal foundation model conditioned on demographic metadata.","marker":"[9]"},{"why":"Provides FLAIR, the vision-language retinal model whose CLIP-style training the paper tests.","marker":"[15]"},{"why":"Provides URFound, the multimodal retinal model that performs best across the paper's evaluations.","marker":"[21]"},{"why":"Supplies the pretrained VAE whose latent space defines the conventional latent-diffusion baseline.","marker":"[14]"},{"why":"Supplies the UK Biobank colour fundus images and linked phenotype labels used for training and evaluation.","marker":"[1]"},{"why":"Defines the flow-matching objective used in both stages of the representation-tokenizer training.","marker":"[10]"}],"fun_headline_variants":["Synthetic retinas: own model says yes, real classifiers say no","Foundation-model retinas fool their own, not real classifiers","Retinal synthesis self-passes, but real-image checks fail","Retinal image synthesis: strong self-evaluation, real-world shortfall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The external ResNet32 classifiers trained only on real images are treated as an unbiased sensor of phenotype preservation in synthetic images, even though the paper itself shows these classifiers lose about 20 percentage points of age explained variance on reconstructed real images.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic retinas: own model says yes, real classifiers say no","Foundation-model retinas fool their own, not real classifiers","Retinal synthesis self-passes, but real-image checks fail","Retinal image synthesis: strong self-evaluation, real-world shortfall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3385,"prompt_tokens":842,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2469}},"tokens_in":458,"tokens_out":2543,"duration_ms":17333,"temperature":1.0,"reasoning_tokens":2469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:36:00.782625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the external classifiers on real images that have been passed through the same reconstruction pipeline (or an equivalent image-to-image transform), then re-run age, sex, and hypertension predictions on synthetic images; if the age $R^2$ gap shrinks to the same size as the drop on reconstructed real images, the synthetic-to-real gap is largely an artifact of classifier sensitivity to transformations rather than a genuine loss of phenotype information.","supporting_citations":[{"cited_title":"et al.: PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning","cited_arxiv_id":null,"evidence_quote":"Provides PRETI, a patient-aware retinal foundation model conditioned on demographic metadata."},{"cited_title":"et al.: A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision","cited_arxiv_id":null,"evidence_quote":"Provides FLAIR, the vision-language retinal model whose CLIP-style training the paper tests."},{"cited_title":"et al.: UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling","cited_arxiv_id":null,"evidence_quote":"Provides URFound, the multimodal retinal model that performs best across the paper's evaluations."}],"review_version":1}