REVIEW 4 major objections 4 minor 27 references
Improving Medical Image Generative Models with Fr\'echet Distance Loss
T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Finetuning medical image generators with Fréchet Distance loss improves downstream tumor segmentation.
desk verdict Useful medical-gen extension of FD-loss, but the headline >5% DSC gain depends on encoders selected on the same test set used for the final numbers; worth refereeing with a corrected protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is the Fréchet distance computed between real images and a queue of generated images in a frozen pretrained feature space. Real-image statistics are computed once and held fixed; generated-image statistics come from a fixed-size queue of 5,000 synthetic-image features that is refreshed each iteration with 48 new samples. Only 16 of those samples are generated with gradient tracking through the 25-step DDIM trajectory, so the population size needed for stable Fréchet estimation is decoupled from the batch size used for gradients, making FD practical as a finetuning loss for small medical datasets.
What would settle it
Run the same FD-loss finetuning but replace the 5,000-image queue with a single large minibatch of fully gradient-tracked generated images (large enough for stable Fréchet statistics), and compare downstream tumor Dice. If the improvement disappears or reverses, the reported gains are an artifact of the queue/gradient approximation rather than of matching feature distributions; if it persists, the Fréchet objective itself is doing the work. A second check is to evaluate FD in a held-out encoder space not used during finetuning: if the real-vs-generated distance only drops in the optimized spac
Extended reading notes
Core claim
The central claim is that FD-loss is an effective regularizer for medical image generative models: finetuning a pretrained segmentation-guided diffusion model with a loss that aligns first- and second-order feature statistics in a frozen encoder space makes generated images more faithful to real tumor morphology and more useful for training downstream segmentation models. The strongest quantitative evidence is the LiTS liver-tumor result, where downstream tumor DSC rises to 52.3% with FD-regularized synthetic augmentation versus 46.9% with unregularized synthetic augmentation alone. The paper further claims that the best encoder space is dataset dependent, with MedDINOv3 optimal for abdomina
Load-bearing premise
The method's gains rest on the assumption that gradients obtained by backpropagating through a 25-step accelerated diffusion sampling trajectory, using only 16 gradient-tracked samples per iteration and queue statistics from 5,000 images, give a stable and unbiased estimate of the true Fréchet distance; if that gradient signal is noisy or biased, the finetuning may not improve the synthetic images.
Editorial extensions
If this is right
- If FD-loss is correct, synthetic images from finetuned generators score closer to real images on FID, KID, CMMD, and FRD across CT and MRI datasets.
- Downstream segmentation networks trained with FD-regularized synthetic data outperform those trained with geometric augmentation or unregularized synthetic data, with the largest gains in heterogeneous tumor regions.
- Encoder choice is a real design lever: matching statistics in MedDINOv3 space helps most for abdominal CT, while BiomedCLIP space helps most for brain MRI.
- Fréchet distance is a better finetuning objective than mean-only, covariance-only, symmetric KL divergence, or Bhattacharyya distance in the tested setting.
- Qualitative evidence links the improvement to more faithful tumor synthesis and fewer segmentation hallucinations.
Reading between the lines
- The same queue-based FD-loss recipe could transfer to latent diffusion or GAN-based medical generators wherever per-pixel objectives smooth high-variance structures, as long as a frozen, domain-appropriate encoder is available.
- Because the best encoder is dataset dependent, a practical guideline is to select the encoder by downstream segmentation performance on a held-out split rather than by image-quality metrics alone.
- A testable extension is to measure whether FD-regularized generators remain superior when sampled with fewer DDIM steps, which would indicate that the regularizer is stabilizing the sample trajectory rather than merely overfitting the training-time sampling schedule.
- FD-loss could serve as an explicit distribution-alignment tool for multi-center or cross-modality harmonization, since it directly minimizes feature-statistic mismatch that often drives domain shift.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes using Fréchet Distance (FD) as a finetuning loss for segmentation-guided diffusion models in medical imaging. Real-image statistics are precomputed in a frozen pretrained encoder space; synthetic-image statistics are estimated from a queue of 5,000 generated images; and the generator is finetuned by backpropagating through 25-step DDIM trajectories. The method is evaluated on LiTS (liver CT), MCT-LTDiag (liver CT), and BraTS (brain MRI) by measuring synthetic image quality (FID, KID, IS, CMMD, FRD) and downstream U-Net segmentation after synthetic-data augmentation. The central claim is that FD-regularized synthetic augmentation improves tumor DSC by more than 5% over unregularized synthetic augmentation alone.
Significance. An effective FD-loss regularizer would be a practical contribution to medical image generation, where per-pixel diffusion losses are known to smooth heterogeneous tumor morphology. The paper has useful ingredients: a broad multi-dataset, multi-modality evaluation; multiple pretrained encoders; a distance-function ablation; and downstream segmentation as the primary endpoint. The ensembled U-Net evaluation is also a plus. However, the headline quantitative claim is currently not established because the encoder for the main comparison is selected on the same held-out test set used for final evaluation, and because no variance or direct significance testing is reported for the Syn+FD versus Syn comparison. These are fixable but central issues.
major comments (4)
- [§4.4 (Table 3), §4.2 (Table 2)] Table 3 and Table 2 report identical DSC/ASSD values for the selected encoders (e.g., LiTS MedDINOv3 87.8/52.3 appears in both tables). The encoder for Tables 1 and 2 is chosen as the best row in Table 3, which is evaluated on the same held-out downstream segmentation set. Thus the reported Syn+FD gain is an optimistic, post-selection estimate, not the performance of a pre-specified method. Please choose encoders on a validation split, pre-specify them, or present Table 3 as the main result and treat Table 2 as illustrative.
- [§4.1–§4.3, Table 2] Table 2 reports no standard deviations/confidence intervals, despite training five U-Nets per configuration. The significance markers only indicate tests against 'no augmentation' and 'geometric augmentation'; no test is reported between Syn.+FD and Syn., which is the comparison highlighted in the abstract. Without error bars and a pairwise significance test, the central claim of improvement over unregularized synthetic augmentation is not statistically supported.
- [Abstract, §5] The abstract and conclusion state that FD-regularized synthetic data 'consistently' improve tumor DSC by '>5%' over unregularized synthetic augmentation alone. Inspecting Table 2, only LiTS liver tumor meets this threshold (46.9→52.3); MCT improves by 3.6 points and BraTS by 1.0–1.9 points. The >5% claim is therefore dataset-specific, not general, and the 'consistently' phrasing overstates the results. Please revise the claim or restrict it to LiTS.
- [§3.1, §4.4] The FD-loss pipeline introduces several unablated hyperparameters (queue size N=5,000, gradient-tracked batch size M=16, 25 DDIM steps, finetuning learning rate 1e-7, 20,000 steps, and the Tikhonov epsilon in Table 4). Because the FD gradient is estimated from a small tracked batch and a finite queue, some sensitivity analysis or at least convergence curves for the FD-loss trajectory are needed to confirm that the reported gains are attributable to FD-loss rather than to a particular hyperparameter configuration.
minor comments (4)
- [Eq. (1)] The Fréchet distance is written as Tr(Σr + Σg − 2(ΣrΣg)^{1/2}); since ΣrΣg is not symmetric in general, please specify the matrix square-root convention or use the standard symmetric form Tr(Σr + Σg − 2(Σr^{1/2}ΣgΣr^{1/2})^{1/2}).
- [Table 4] The rows FDµOnly and FDΣOnly are not defined in the text; state whether they set the covariance or mean term to zero/identity.
- [Fig. 2] The figure is difficult to read: DSC numbers are small, and the encoder labels ('MedDINOv3 / MM-DINOv2') conflate two model names. Add per-panel captions and larger text.
- [General] No code or data availability statement is included; given the number of hyperparameters and seeds, a reproducibility statement would be valuable.
Circularity Check
No equation-level circularity; the headline DSC gain may be inflated by test-set encoder selection, but that is selection bias, not circularity.
full rationale
The derivation chain is self-contained. FD-loss (Eq. 1) is a training objective computed from precomputed real-image statistics and a queue of generated-image features in a frozen encoder; the paper's reported claims are downstream tumor DSC on held-out real images, plus FID/KID/IS/CMMD/FRD. None of these evaluation quantities equals the optimized objective by construction: the FD-loss encoder space (MedDINOv3/BiomedCLIP) differs from the feature spaces used for FID/KID/IS/CMMD, and DSC is a clinical segmentation metric on real images. The only self-citation, FRD [10], is used as an external metric and as motivation for a correlation with segmentation; the experimental conclusions do not reduce to it, so it is not load-bearing. A real protocol concern -- not an equation-level circularity -- is that Section 4.4 selects the per-dataset encoder on the same held-out downstream test set later used in Table 2 ('As shown in Table 3, the optimal representation is dataset dependent: MedDINOv3 achieves the best overall performance on the abdominal CT datasets, LiTS and MCT-LTDiag, whereas BiomedCLIP performs best on the BraTS brain MRI dataset'); Table 2 then reports Syn+FD with exactly these winners. This can optimistically bias the abstract's >5% DSC claim, but it is test-set selection bias / missing an independent validation split, not a reduction of the prediction to its inputs. Under the stated circularity definition, the score is 2.
Assumptions & free parameters
free parameters (8)
- Feature-extractor selection per dataset =
MedDINOv3 (liver CT); BiomedCLIP (BraTS)
- Queue size N =
5,000
- Gradient-tracked batch size M =
16
- Non-gradient images per iteration =
32
- Finetuning learning rate =
1e-7
- Finetuning steps =
20,000
- DDIM sampling steps =
25
- Tikhonov epsilon =
1e-6
assumptions (6)
- domain assumption Diffusion model with velocity parameterization and mask conditioning is a valid base generator.
- standard math Feature distributions are well-approximated by multivariate Gaussians.
- domain assumption Pretrained encoders (InceptionV3, RadInceptionV3, BiomedCLIP, MedDINOv3, MM-DINOv2) provide feature spaces whose statistics correlate with clinically relevant tumor morphology.
- domain assumption 25-step DDIM sampling is differentiable and its gradients are a faithful training signal for the FD loss.
- domain assumption Fixed seeds and a five-model soft-voting ensemble adequately control variance for the reported comparisons.
- domain assumption Queue-based estimation with N=5000 generated images and incremental updates approximates the true generated distribution well enough for FD minimization.
Cite this review
Pith. "Pith review of Improving Medical Image Generative Models with Fr\'echet Distance Loss." pith.science (2026). https://pith.science/paper/D26KXACX
@misc{pith2026260713300,
author = {Pith},
title = {Pith review of: Improving Medical Image Generative Models with Fr\'echet Distance Loss},
year = {2026},
howpublished = {\url{https://pith.science/paper/D26KXACX}},
note = {Machine review of arXiv:2607.13300}
}
abstract
Diffusion generative models have demonstrated immense potential for synthetic medical image generation. However, these models often struggle to capture complex morphological characteristics of heterogeneous tumors with irregular boundaries, limiting their utility for downstream clinical tasks such as segmentation. This limitation stems from the standard denoising objective: minimizing a per-pixel error, which smooths high-variance irregular structures characteristic of tumors. To address this, we propose finetuning these generative models with Fr\'echet Distance loss (FD-loss). FD-loss aligns the first and second order feature statistics of real and generated images in a pretrained encoder space, encouraging the generator to capture complex structural variations characteristic of heterogeneous tumors. We integrate FD-loss across diverse architectural settings, using both natural- and medical-image encoders on multiple liver and brain cancer datasets spanning CT and MRI modalities. Downstream segmentation networks trained on our FD-regularized synthetic data consistently achieve superior performance, improving tumor DSC by $>$$5\%$ over unregularized synthetic augmentation alone. Qualitative analysis suggests these gains are associated with more faithful tumor synthesis and fewer segmentation hallucinations. Our results show FD-loss as an effective regularizer for medical image generative models to improve clinical workflows.
Figures
Reference graph
Works this paper leans on
-
[1]
Scientific Data4, 170117 (2017).https://doi.org/10.1038/sdata.2017.117
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., et al.: Advancing the cancer genome atlas glioma MRI col- lections with expert segmentation labels and radiomic features. Scientific Data4, 170117 (2017).https://doi.org/10.1038/sdata.2017.117
-
[2]
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R.T., Berger, C., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge (2019).https://doi.org/10.48550/arXiv.1811.02629
-
[3]
Medical Image Analysis84, 102680 (2023).https://doi.org/10.1016/j.media.2022.102680
Bilic,P.,Christ,P.,Li,H.B.,Vorontsov,E.,Ben-Cohen,A.,Kaissis,G.,Szeskin,A., Jacobs, C., et al.: The liver tumor segmentation benchmark (LiTS). Medical Image Analysis84, 102680 (2023).https://doi.org/10.1016/j.media.2022.102680
arXiv 2023
-
[4]
Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying MMD GANs (2021).https://doi.org/10.48550/arXiv.1801.01401
-
[5]
Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., et al.: MONAI: An open-source framework for deep learning in healthcare (2022).https://doi.org/10.48550/arXiv.2211.02701
-
[6]
https://doi.org/10.48550/arXiv.1706.08500
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium (2018). https://doi.org/10.48550/arXiv.1706.08500
-
[7]
In: Advances in Neural Information Processing Systems
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems. vol. 33, pp. 6840– 6851 (2020),https://proceedings.neurips.cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf
2020
-
[8]
In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Jayasumana,S.,Ramalingam,S.,Veit,A.,Glasner,D.,Chakrabarti,A.,Kumar,S.: Rethinking FID: Towards a better evaluation metric for image generation. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9307–9315. IEEE (2024).https://doi.org/10.1109/CVPR52733.2024.00889
arXiv 2024
Show all 27 references
- [9]
-
[10]
Medical Image Analysis110, 103943 (2026)
Konz, N., Osuala, R., Verma, P., Chen, Y., Gu, H., Dong, H., Chen, Y., Mar- shall, A., et al.: Fréchet radiomic distance (FRD): A versatile metric for com- paring medical imaging datasets. Medical Image Analysis110, 103943 (2026). https://doi.org/10.1016/j.media.2026.103943
2026
-
[11]
Li, Y., Wu, Y., Lai, Y., Hu, M., Yang, X.: MedDINOv3: How to adapt vision foundation models for medical image segmentation? (2025).https://doi.org/ 10.48550/arXiv.2509.02379
2025 doi
- [12]
- [13]
-
[14]
Radiology: Artificial Intelligence4(5), e210315 (2022).https://doi.org/10.1148/ryai.210315
Mei, X., Liu, Z., Robson, P.M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K.E., et al.: RadImageNet: An open radiologic deep learning re- search dataset for effective transfer learning. Radiology: Artificial Intelligence4(5), e210315 (2022).https://doi.or...
2022 doi
-
[15]
IEEE transactions on medical imaging34(10), 1993–2024 (2015).https://doi.org/10.1109/TMI.2014.2377694
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging34(10), 1993–2024 (2015).https://doi.org/10.1109/TMI.2014.2377694
1993
- [16]
- [17]
- [18]
- [19]
- [20]
-
[21]
In: 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the in- ception architecture for computer vision. In: 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR). pp. 2818–2826. IEEE (2016). https://doi.org/10.1109/CVPR.2016.308
2016 doi
-
[22]
In: Medical Image Computing and Computer Assisted Intervention – MICCAI
Woodland, M., Castelo, A., Al Taie, M., Albuquerque Marques Silva, J., Elta- her, M., Mohn, F., Shieh, A., Kundu, S., et al.: Feature extraction for gen- erative medical imaging evaluation: New evidence against an evolving trend. In: Medical Image Computing and Computer Assist...
-
[23]
Sci- entific Data13(1), 31 (2025).https://doi.org/10.1038/s41597-025-06343-4
Wu, X., Su, H., Hua, Y., Xu, Y., Wang, L., Wang, X., Wang, S., Jin, B., et al.: A multi-phase CT dataset for automated differential diagnosis of liver tumors. Sci- entific Data13(1), 31 (2025).https://doi.org/10.1038/s41597-025-06343-4
2025 doi
- [24]
-
[25]
Nature Communications16(1), 6486 (2025).https://doi.org/10.1038/ s41467-025-61754-6 Improving Medical Image Generative Models with Fréchet Distance Loss 11
Zhang, L., Jindal, B., Alaa, A., Weinreb, R., Wilson, D., Segal, E., Zou, J., Xie, P.: Generative AI enables medical image segmentation in ultra low-data regimes. Nature Communications16(1), 6486 (2025).https://doi.org/10.1038/ s41467-025-61754-6 Improving Medical Image Genera...
2025
- [26]
-
[2024]
pp. 87–97. Springer Nature Switzerland (2024).https://doi.org/10.1007/ 978-3-031-72390-2_9
2024
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.