{"id":"ebf1da9a-4d70-4f28-a846-d4af59106d30","arxiv_id":"2506.11753","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Retinal image synthesis with VQ-GAN is not improved by a RETFound-based deep feature loss, and a simple edge-detection loss performs competitively.","lead":"This paper tests whether a loss function built from domain-specific foundation model features improves synthetic retinal image generation. It finds that they do not, and that a simple edge-detection loss matches more complex perceptual losses at lower cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RETFound loss uses one final ViT layer while perceptual loss sums five VGG layers and loss weights are unreported, so the negative claim is confounded and over-generalized.","rationale":"The reader's weakest_assumption correctly identifies the layer-count confound between RETFound loss and perceptual loss. My stress-test agrees that this is the most load-bearing issue, but sharpens it: the comparison also varies architecture, distance norm, and, critically, unreported loss weights. The manuscript itself admits the layer-count and architecture alternatives in Section 5, which strengthens the concern and should be weighed in the verdict. The reader's CONDITIONAL verdict already reflects this uncertainty, so no verdict change is needed. I also note a smaller internal inconsistency: the text in Section 4.2 says no synthetic model achieved error rates as low as real data, but Table 2 shows VQ-GAN (Perceptual) achieving a lower MAE for age (7.60 ± 0.27) than real data (7.72 ± 0.13). This overstatement is secondary to the main confound but reinforces the need for careful wording. Overall, the paper is a useful empirical study with a clear configuration-specific finding, but the broad central claim is not fully supported until the ablation described above is performed.","tokens_in":9441,"tokens_out":2919,"duration_ms":30277,"concrete_test":"Run a matched ablation with all loss weights explicitly fixed and equal (e.g., λ = 1) for four VQ-GAN variants: (a) RETFound final-layer-only L1, (b) RETFound sum-of-five-layers L1, (c) VGG-16 final-layer-only L2, and (d) VGG-16 sum-of-five-layers L2. Compare FID and the number of AutoMorph features passing the permutation test. If (b) matches or beats (d), the claim that domain-specific features do not help is refuted; if (b) remains worse than (d), the main confound is resolved in the authors' favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative claim, stated in Section 5, is that deep features from a domain-specific foundation model do not yield more realistic or morphologically accurate synthetic retinal images. The evidence for this claim rests on comparing VQ-GAN (RETFound), trained with the RETFound loss of Section 3.3, against VQ-GAN (Perceptual), trained with the perceptual loss of Section 3.5. These two losses differ along multiple axes at once: (1) feature source, RETFound's self-supervised retinal ViT vs VGG-16 trained on ImageNet; (2) architecture, a vision transformer vs a CNN; (3) layer aggregation, one final 1024-dimensional embedding layer vs a sum of five activation layers (4th, 9th, 16th, 23rd, 30th); and (4) the distance norm, L1 for RETFound vs L2 for VGG. Loss weights are not reported, so the total gradient contribution of the RETFound term relative to pixel and GAN losses is unknown. The paper's own Discussion explicitly flags the layer-count difference as 'another possible reason' for the relative drop in RETFound loss performance, and the architecture difference is also acknowledged. Thus the evidence establishes only that this particular single-layer, weight-unreported RETFound loss did not help; it does not establish the broad conclusion that domain-specific deep features are ineffective. This is load-bearing because the headline contribution is precisely that negative general claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether a perceptual-style loss built on deep features of RETFound, a self-supervised foundation model trained on 1.6 million colour fundus images, improves VQ-GAN-based synthesis of retinal images compared with a VGG-16 perceptual loss, an edge-detection (Meijering filter) loss, and a pixel-only baseline. All models are trained on 90,344 UK Biobank colour fundus images at 256x256 resolution and evaluated on FID/MMD/MS-SSIM, on Train-Synthetic/Test-Real prediction of age, sex, BMI, blood pressure, HbA1c and glucose using AutoMorph-extracted features, and on distributional matching of 45 retinal morphology features via permutation tests. The authors conclude in Section 5 that deep features from a domain-specific foundation model do not yield more realistic or morphologically accurate synthetic fundus images, and that a conventional edge-based loss is a cheap and competitive alternative.","tokens_in":9665,"tokens_out":8515,"duration_ms":71023,"significance":"If established, the negative result is a useful contribution to medical image synthesis: it challenges the implicit assumption that domain-specific foundation-model features are the right backbone for perceptual-style losses in this setting, and it identifies edge-based supervision as a practical, low-cost alternative. The study has genuine methodological strengths: the evaluation pipeline is unusually comprehensive for the area, spanning fidelity, downstream clinical prediction, and direct morphological comparison; the evaluation metrics and feature extractors (AutoMorph, RETFound, InceptionV3) are external pretrained models independent of the training losses, so circularity is minimal; and the central claim is a clearly falsifiable negative result rather than another positive demonstration on a single metric. The main weakness is that the headline claim is stated more broadly than the experiments can support: the RETFound loss and the perceptual loss differ in architecture, layer count, distance norm, resolution, and unreported loss weights, and the paper's own Discussion concedes that these factors are plausible explanations for the observed gap.","major_comments":[{"comment":"The central negative claim that domain-specific deep features are ineffective rests on comparing VQ-GAN (RETFound) with VQ-GAN (Perceptual), but the two losses differ along several axes at once. The RETFound loss (Section 3.3) is an L1 distance on a single 1024-dimensional final-layer embedding of a ViT-Large, computed after resizing inputs to 224x224, whereas the perceptual loss (Section 3.5) is a sum of L2 distances over five VGG-16 activation layers (4th, 9th, 16th, 23rd, 30th) at the native 256x256 resolution. Feature source, architecture (ViT vs CNN), layer count (1 vs 5), distance norm (L1 vs L2), and resolution are therefore all varied simultaneously, and the weighting of the auxiliary loss terms relative to the pixel and GAN losses is never reported (Section 2.1 lists three gradient sources without coefficients). The Discussion (Section 5) itself names the architecture difference and the layer-count difference as possible reasons for the RETFound loss underperforming, which concedes that the experiment does not isolate the effect of feature source. As stated, the headline conclusion overgeneralizes from a single confounded configuration; the authors should either add ablations that match layer count and architecture (for example, a multi-layer RETFound loss or a single-layer VGG loss) or narrow the claim to the specific loss configuration tested.","section":"§2.1, §3.3, §3.5, §5"},{"comment":"The sentence in Section 4.2 that 'none of the synthetic models achieved error rates as low as those based on real data' is contradicted by the paper's own Table 2: for age, VQ-GAN (Perceptual) reports MAE 7.60 (0.27), which is lower than the real-data MAE of 7.72 (0.13); for sex, VQ-GAN (RETFound) reports F1 0.53 (0.01), higher than the real-data 0.52 (0.01). The blanket statement is therefore factually incorrect as written and should be replaced with a per-variable comparison or a qualified statement such as 'in most prediction tasks.'","section":"§4.2, Table 2"},{"comment":"FID and MMD are reported in Table 1 as single point values with no error bars, and the number of training runs or evaluation seeds is not stated anywhere. For a paper whose main contribution is a negative result, the absence of uncertainty quantification is load-bearing: the FID spread across models (2.45, 5.47, 11.43, 19.95) and the morphological 'matches' counted in Section 4.3 (3, 3, 4, and 5 of 45 features) are small-number comparisons, and without confidence intervals, multiple seeds, or a statistical test the reader cannot judge whether the ordering of the models is stable. This is especially important for the claim that the RETFound loss does not improve over the pixel-only baseline, since its FID of 11.43 is closer to real data than the pixel baseline's 19.95 but this difference may be within run-to-run noise.","section":"§4.1, Tables 1–2, §4.3"},{"comment":"The Discussion states that 'the RETFound loss consistently underperformed,' but Table 1 shows that VQ-GAN (RETFound) achieves FID 11.43 versus 19.95 for VQ-GAN (Pixel), i.e., the RETFound loss substantially improves fidelity relative to the pixel-only baseline on this metric. The abstract's claim that domain-specific deep features 'do not improve autoencoder image generation' is likewise in tension with the paper's own evidence: the RETFound loss improves FID over the baseline while worsening downstream-task performance and morphological matching. The paper's conclusions should be reframed as a mixed, metric-dependent result, with the specific negative finding being that RETFound features underperform VGG perceptual features and edge-based supervision on most downstream and morphological measures.","section":"§5, Table 1"}],"minor_comments":[{"comment":"Several LaTeX or line-break artifacts remain in the text ('Fr\\'echet', 'in-vestigated', 'autoen-coder'); these should be cleaned up before final submission.","section":"Abstract"},{"comment":"Table 2 uses the label 'VQGAN (RETFound)' while the text and Table 1 use 'VQ-GAN (RETFound)'; the notation should be unified.","section":"Table 2"},{"comment":"Section 3.1 does not state how the 90,344 UK Biobank images were split into training, validation, and test sets, and Section 4.1 does not say whether the 'Real' row in Table 1 is computed on the same 20,000-image subset used for the synthetic comparisons; both details should be reported.","section":"§3.1, §4.1"},{"comment":"The Figure 3 caption claims on the basis of a single sample that VQ-GAN (RETFound) 'struggles to replicate the optic cup'; this qualitative observation should either be tied to the quantitative optic-disc/cup features in Figure 4 or softened.","section":"Fig. 3"},{"comment":"The citation of PeCo [33] to argue that transformer-based perceptual losses outperform pixel-based losses is not directly applicable, since PeCo is a pre-training objective rather than a frozen-feature reconstruction loss; the relevance to the present setup should be explained or the sentence revised.","section":"§5, ref. [33]"}],"recommendation":"major_revision","confidential_remarks":"The paper is a genuinely useful negative result in an area where null findings are rarely reported, but the headline claim needs to be narrowed or the experiments need to be extended. The confound discussed in major comment 1 is the core issue, and the factual contradiction in Section 4.2 must be fixed. The margins between models on the morphological matches are small (3 vs 5 of 45 features), so the authors should be encouraged to add error bars and seeds rather than rely on point estimates. The manuscript would fit a venue that values rigorous negative results and study design; I would not recommend it for a venue requiring a strong positive methodological advance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: useful empirical comparison, but the headline negative claim is broader than the experiment supports. The RETFound loss is compared against perceptual loss with three things different at once—backbone (ViT vs CNN), number of feature layers (one vs five), and norm (L1 vs L2)—and loss weights are not reported. So the result is \"this particular RETFound configuration didn't help,\" not \"domain-specific deep features don't help.\" Interestingly, the authors themselves name the layer-count difference in the Discussion, which is honest, but the abstract and Section 5 still make the general claim.\n\nWhat's genuinely new and good: the paper runs a four-way loss comparison for colour fundus VQ-GAN, and it validates beyond FID: downstream prediction of six clinical variables and direct comparison of 45 AutoMorph retinal morphology features. That is a thorough pipeline, and the finding that an edge-detection loss (Meijering filter, zero learned parameters) is competitive with VGG perceptual loss is practical and reproducible. The negative result for RETFound is plausible directionally—the numbers consistently put it at or near the bottom—so the paper is worth reading for anyone choosing losses for retinal synthesis.\n\nSoft spots, in order: (1) the confound above is load-bearing because the contribution is the negative general claim. (2) Table 2 actually contradicts the text: VQ-GAN (Perceptual) gives lower MAE for age (7.60) than real data (7.72), so \"none of the synthetic models achieved error rates as low as those based on real data\" is false as written. (3) FID and MMD are reported without error bars, and the MMD values barely discriminate. (4) No code release, which matters for a paper whose main message is about a particular implementation.\n\nWho it's for: practitioners working on medical image synthesis who want evidence on loss choices. For them, the edge-loss result is the takeaway. The broader claim about foundation models needs a controlled follow-up.\n\nRecommendation: send to peer review—the question is timely and the evaluation pipeline is substantial—but the authors should be required to match layer counts and loss weights, add uncertainty estimates, fix the Table 2 overstatement, and ideally release code.","headline":"Useful empirical comparison with a practical edge-loss takeaway, but the headline claim about domain-specific deep features is overgeneralized because the RETFound loss is confounded with architecture, layer count, and unreported loss weights.","tokens_in":10221,"tokens_out":2337,"would_cite":true,"duration_ms":22273,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep features from a retinal foundation model do not make synthetic colour fundus images more realistic or clinically accurate, while a simple edge-detection loss nearly matches a natural-image perceptual loss.","keywords":["colour fundus imaging","synthetic medical images","VQ-GAN","perceptual loss","retinal image synthesis","foundation model features","edge detection loss","vascular morphology"],"falsifier":"Retrain the VQ-GAN with a RETFound-based loss that, like the perceptual loss, sums distances over several transformer layers instead of only the final layer. If FID, downstream prediction errors, and feature-overlap p-values then rise to match the perceptual-loss model, the claim that domain-specific deep features do not help would be contradicted.","tokens_in":1419,"feed_emoji":"👁️","tokens_out":1437,"duration_ms":48459,"temperature":0.7,"pith_summary":"This paper asks whether using deep features from a foundation model trained on retinal images improves synthetic colour fundus image generation. The authors train VQ-GAN autoencoders with four supervision losses: pixel-only, edge-based, perceptual loss from a natural-image network, and a new loss computed in the latent space of RETFound, a vision transformer pretrained on 1.6 million retinal images. Across fidelity metrics, downstream cardiovascular risk prediction, and direct comparison of 45 retinal morphology features, the RETFound loss underperforms and does not clearly beat the pixel-only baseline. Perceptual loss leads on most criteria, and a simple Meijering edge-detection loss is nearly as good at far lower cost. The paper concludes that domain-specific foundation-model features do not yield more realistic or clinically accurate synthetic retinal images.","feed_headline":"Domain-specific AI features do not improve synthetic retinal images","feed_subtitle":"Simple edge filters match costly foundation-model losses, while natural-image perceptual loss still leads.","key_machinery":"The key object is the RETFound loss: the L1 distance in the 1024-dimensional embedding space of the last transformer block of RETFound, computed between real and synthetic images and added to a VQ-GAN training scheme alongside pixel and GAN losses. The comparison machinery includes perceptual loss, which sums L2 distances over five VGG-16 activation layers, and edge loss, which computes L2 distance between Meijering-filter vesselness responses on the green channel. Validation runs through FID, MMD, and MS-SSIM for fidelity, downstream Train-Synthetic/Test-Real prediction of cardiovascular risk factors, and permutation tests over 45 AutoMorph retinal morphology features.","core_discovery":"The central claim is that leveraging deep features of foundational models trained on domain data does not yield samples that are more realistic or that better preserve morphological and clinical information in colour fundus images. In head-to-head comparison, the VQ-GAN supervised by RETFound loss achieved FID 11.43 versus 2.45 for perceptual loss and 5.47 for edge loss, and in downstream clinical prediction tasks it performed worse than even the pixel-only baseline. Perceptually supervised models matched the largest share of real retinal features in permutation tests, while the RETFound-supervised model deviated most on fractal dimension and vascular complexity. The authors note that this is surprising given the established use of RETFound encoder features for cardiovascular risk prediction, and they list architectural differences, such as a vision transformer versus a CNN and reliance on a single activation layer, as possible explanations.","pith_inferences":["Editorial inference: the paper does not isolate whether the failure comes from domain-specific features, from the vision-transformer architecture, or from using only one activation layer; a fair test would need a multi-layer RETFound loss or a CNN retinal backbone.","Editorial inference: if the single-layer implementation drives the gap, the broader conclusion that domain-specific features do not help would be too hasty and should be tested directly.","Editorial inference: the strong edge-loss result hints that low-level vessel geometry is the bottleneck for clinical utility, more than high-level semantic features; combining edge loss with perceptual loss could test this.","Editorial inference: the validation design could serve as a template for other medical modalities, reporting feature-overlap and downstream-task performance rather than relying on FID alone."],"forward_implications":["For colour fundus synthesis, a domain-specific foundation-model perceptual loss built from a single final layer is not worth its compute relative to perceptual or edge losses.","Simple edge-detection supervision is a practical, weight-free alternative that nearly matches expensive deep-feature losses for vascular sharpness.","Downstream clinical prediction and morphological feature overlap should be part of evaluating synthetic medical images, since fidelity metrics such as MMD and MS-SSIM alone could not separate the models.","No tested model matched real images in downstream prediction, so synthetic fundus data still cannot replace real data for these tasks.","The result leaves open whether transformer-based losses or multi-layer perceptual losses could improve synthesis, because the RETFound loss used only one activation layer."],"supporting_citations":[{"why":"Supplies RETFound, the domain-specific foundation model whose final-layer latent space defines the proposed loss.","marker":"[3]"},{"why":"Defines perceptual loss, the main comparison baseline that leads on most metrics.","marker":"[25]"},{"why":"Supplies the VQ-GAN architecture and training scheme that all four loss variants modify.","marker":"[10]"},{"why":"Supplies AutoMorph, the feature extractor whose 45 retinal features drive the morphological and downstream evaluation.","marker":"[6]"},{"why":"Supplies the Meijering filter used in the edge-loss vesselness maps.","marker":"[29]"},{"why":"Establishes the cardiovascular risk-factor prediction tasks used to evaluate clinical relevance of synthetic images.","marker":"[1]"},{"why":"Provides the Train-Synthetic/Test-Real evaluation protocol used for downstream validation.","marker":"[32]"}],"fun_headline_variants":["Edge filters outperform domain-specific AI features in retinal synthesis","Perceptual loss still beats domain-specific deep features for retinal images","Simple edge detection surpasses foundation-model losses in retinal synthesis","Domain-specific deep features fail to boost realism in synthetic retinal images"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The comparison assumes that the two losses differ mainly in whether the features come from a retinal foundation model or a natural-image network; in fact they also differ in number of network layers, so the disappointing results may come from the implementation rather than from domain-specific features.","fun_headline_variants_meta":{"raw":{"variants":["Edge filters outperform domain-specific AI features in retinal synthesis","Perceptual loss still beats domain-specific deep features for retinal images","Simple edge detection surpasses foundation-model losses in retinal synthesis","Domain-specific deep features fail to boost realism in synthetic retinal images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2619,"prompt_tokens":912,"completion_tokens":1707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1639}},"tokens_in":528,"tokens_out":1707,"duration_ms":15404,"temperature":1.0,"reasoning_tokens":1639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:44.725585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the VQ-GAN with a RETFound-based loss that, like the perceptual loss, sums distances over several transformer layers instead of only the final layer. If FID, downstream prediction errors, and feature-overlap p-values then rise to match the perceptual-loss model, the claim that domain-specific deep features do not help would be contradicted.","supporting_citations":[{"cited_title":"Poplin et al., ‘Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning’, Nat Biomed Eng, vol","cited_arxiv_id":null,"evidence_quote":"Establishes the cardiovascular risk-factor prediction tasks used to evaluate clinical relevance of synthetic images."},{"cited_title":"Jordon et al., ‘Synthetic Data -- what, why and how?’, May 2022","cited_arxiv_id":null,"evidence_quote":"Provides the Train-Synthetic/Test-Real evaluation protocol used for downstream validation."}],"review_version":1}