Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Improving Medical Image Generative Models with Fr\'echet Distance Loss

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read Finetuning medical image generators with Fréchet Distance loss improves downstream tumor segmentation.

desk verdict Useful medical-gen extension of FD-loss, but the headline >5% DSC gain depends on encoders selected on the same test set used for the final numbers; worth refereeing with a corrected protocol. read the letter →

arxiv 2607.13300 v1 pith:D26KXACX submitted 2026-07-14 cs.CV

classification cs.CV
keywords Fréchetdistancelossmedicalimagegenerationdiffusionmodelssegmentation-guidedtumorsegmentationsyntheticdataaugmentationDicesimilaritycoefficientfeaturedistributionmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard denoising objective used to train medical diffusion models, a per-pixel error, tends to smooth the irregular, high-variance boundaries of heterogeneous tumors, which makes synthetic images less useful for training segmentation networks. To fix this, it proposes finetuning a pretrained segmentation-guided generator with Fréchet Distance loss (FD-loss), which matches the mean and covariance of real and generated images in a frozen pretrained feature space. On liver CT and brain MRI datasets, FD-regularized synthetic augmentation improves downstream tumor segmentation by more than 5 percentage points in Dice score over unregularized synthetic data. The paper also shows that the choice of feature extractor matters, with domain-specific medical encoders generally outperforming the natural-image encoder.

What carries the argument

The load-bearing device is the Fréchet distance computed between real images and a queue of generated images in a frozen pretrained feature space. Real-image statistics are computed once and held fixed; generated-image statistics come from a fixed-size queue of 5,000 synthetic-image features that is refreshed each iteration with 48 new samples. Only 16 of those samples are generated with gradient tracking through the 25-step DDIM trajectory, so the population size needed for stable Fréchet estimation is decoupled from the batch size used for gradients, making FD practical as a finetuning loss for small medical datasets.

What would settle it

Run the same FD-loss finetuning but replace the 5,000-image queue with a single large minibatch of fully gradient-tracked generated images (large enough for stable Fréchet statistics), and compare downstream tumor Dice. If the improvement disappears or reverses, the reported gains are an artifact of the queue/gradient approximation rather than of matching feature distributions; if it persists, the Fréchet objective itself is doing the work. A second check is to evaluate FD in a held-out encoder space not used during finetuning: if the real-vs-generated distance only drops in the optimized spac

Watch

Extended reading notes

Core claim

The central claim is that FD-loss is an effective regularizer for medical image generative models: finetuning a pretrained segmentation-guided diffusion model with a loss that aligns first- and second-order feature statistics in a frozen encoder space makes generated images more faithful to real tumor morphology and more useful for training downstream segmentation models. The strongest quantitative evidence is the LiTS liver-tumor result, where downstream tumor DSC rises to 52.3% with FD-regularized synthetic augmentation versus 46.9% with unregularized synthetic augmentation alone. The paper further claims that the best encoder space is dataset dependent, with MedDINOv3 optimal for abdomina

Load-bearing premise

The method's gains rest on the assumption that gradients obtained by backpropagating through a 25-step accelerated diffusion sampling trajectory, using only 16 gradient-tracked samples per iteration and queue statistics from 5,000 images, give a stable and unbiased estimate of the true Fréchet distance; if that gradient signal is noisy or biased, the finetuning may not improve the synthetic images.

Editorial extensions

If this is right

  • If FD-loss is correct, synthetic images from finetuned generators score closer to real images on FID, KID, CMMD, and FRD across CT and MRI datasets.
  • Downstream segmentation networks trained with FD-regularized synthetic data outperform those trained with geometric augmentation or unregularized synthetic data, with the largest gains in heterogeneous tumor regions.
  • Encoder choice is a real design lever: matching statistics in MedDINOv3 space helps most for abdominal CT, while BiomedCLIP space helps most for brain MRI.
  • Fréchet distance is a better finetuning objective than mean-only, covariance-only, symmetric KL divergence, or Bhattacharyya distance in the tested setting.
  • Qualitative evidence links the improvement to more faithful tumor synthesis and fewer segmentation hallucinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same queue-based FD-loss recipe could transfer to latent diffusion or GAN-based medical generators wherever per-pixel objectives smooth high-variance structures, as long as a frozen, domain-appropriate encoder is available.
  • Because the best encoder is dataset dependent, a practical guideline is to select the encoder by downstream segmentation performance on a held-out split rather than by image-quality metrics alone.
  • A testable extension is to measure whether FD-regularized generators remain superior when sampled with fewer DDIM steps, which would indicate that the regularizer is stabilizing the sample trajectory rather than merely overfitting the training-time sampling schedule.
  • FD-loss could serve as an explicit distribution-alignment tool for multi-center or cross-modality harmonization, since it directly minimizes feature-statistic mismatch that often drives domain shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes using Fréchet Distance (FD) as a finetuning loss for segmentation-guided diffusion models in medical imaging. Real-image statistics are precomputed in a frozen pretrained encoder space; synthetic-image statistics are estimated from a queue of 5,000 generated images; and the generator is finetuned by backpropagating through 25-step DDIM trajectories. The method is evaluated on LiTS (liver CT), MCT-LTDiag (liver CT), and BraTS (brain MRI) by measuring synthetic image quality (FID, KID, IS, CMMD, FRD) and downstream U-Net segmentation after synthetic-data augmentation. The central claim is that FD-regularized synthetic augmentation improves tumor DSC by more than 5% over unregularized synthetic augmentation alone.

Significance. An effective FD-loss regularizer would be a practical contribution to medical image generation, where per-pixel diffusion losses are known to smooth heterogeneous tumor morphology. The paper has useful ingredients: a broad multi-dataset, multi-modality evaluation; multiple pretrained encoders; a distance-function ablation; and downstream segmentation as the primary endpoint. The ensembled U-Net evaluation is also a plus. However, the headline quantitative claim is currently not established because the encoder for the main comparison is selected on the same held-out test set used for final evaluation, and because no variance or direct significance testing is reported for the Syn+FD versus Syn comparison. These are fixable but central issues.

major comments (4)
  1. [§4.4 (Table 3), §4.2 (Table 2)] Table 3 and Table 2 report identical DSC/ASSD values for the selected encoders (e.g., LiTS MedDINOv3 87.8/52.3 appears in both tables). The encoder for Tables 1 and 2 is chosen as the best row in Table 3, which is evaluated on the same held-out downstream segmentation set. Thus the reported Syn+FD gain is an optimistic, post-selection estimate, not the performance of a pre-specified method. Please choose encoders on a validation split, pre-specify them, or present Table 3 as the main result and treat Table 2 as illustrative.
  2. [§4.1–§4.3, Table 2] Table 2 reports no standard deviations/confidence intervals, despite training five U-Nets per configuration. The significance markers only indicate tests against 'no augmentation' and 'geometric augmentation'; no test is reported between Syn.+FD and Syn., which is the comparison highlighted in the abstract. Without error bars and a pairwise significance test, the central claim of improvement over unregularized synthetic augmentation is not statistically supported.
  3. [Abstract, §5] The abstract and conclusion state that FD-regularized synthetic data 'consistently' improve tumor DSC by '>5%' over unregularized synthetic augmentation alone. Inspecting Table 2, only LiTS liver tumor meets this threshold (46.9→52.3); MCT improves by 3.6 points and BraTS by 1.0–1.9 points. The >5% claim is therefore dataset-specific, not general, and the 'consistently' phrasing overstates the results. Please revise the claim or restrict it to LiTS.
  4. [§3.1, §4.4] The FD-loss pipeline introduces several unablated hyperparameters (queue size N=5,000, gradient-tracked batch size M=16, 25 DDIM steps, finetuning learning rate 1e-7, 20,000 steps, and the Tikhonov epsilon in Table 4). Because the FD gradient is estimated from a small tracked batch and a finite queue, some sensitivity analysis or at least convergence curves for the FD-loss trajectory are needed to confirm that the reported gains are attributable to FD-loss rather than to a particular hyperparameter configuration.
minor comments (4)
  1. [Eq. (1)] The Fréchet distance is written as Tr(Σr + Σg − 2(ΣrΣg)^{1/2}); since ΣrΣg is not symmetric in general, please specify the matrix square-root convention or use the standard symmetric form Tr(Σr + Σg − 2(Σr^{1/2}ΣgΣr^{1/2})^{1/2}).
  2. [Table 4] The rows FDµOnly and FDΣOnly are not defined in the text; state whether they set the covariance or mean term to zero/identity.
  3. [Fig. 2] The figure is difficult to read: DSC numbers are small, and the encoder labels ('MedDINOv3 / MM-DINOv2') conflate two model names. Add per-panel captions and larger text.
  4. [General] No code or data availability statement is included; given the number of hyperparameters and seeds, a reproducibility statement would be valuable.

Circularity Check

0 steps flagged · score 2.0 of 10

No equation-level circularity; the headline DSC gain may be inflated by test-set encoder selection, but that is selection bias, not circularity.

full rationale

The derivation chain is self-contained. FD-loss (Eq. 1) is a training objective computed from precomputed real-image statistics and a queue of generated-image features in a frozen encoder; the paper's reported claims are downstream tumor DSC on held-out real images, plus FID/KID/IS/CMMD/FRD. None of these evaluation quantities equals the optimized objective by construction: the FD-loss encoder space (MedDINOv3/BiomedCLIP) differs from the feature spaces used for FID/KID/IS/CMMD, and DSC is a clinical segmentation metric on real images. The only self-citation, FRD [10], is used as an external metric and as motivation for a correlation with segmentation; the experimental conclusions do not reduce to it, so it is not load-bearing. A real protocol concern -- not an equation-level circularity -- is that Section 4.4 selects the per-dataset encoder on the same held-out downstream test set later used in Table 2 ('As shown in Table 3, the optimal representation is dataset dependent: MedDINOv3 achieves the best overall performance on the abdominal CT datasets, LiTS and MCT-LTDiag, whereas BiomedCLIP performs best on the BraTS brain MRI dataset'); Table 2 then reports Syn+FD with exactly these winners. This can optimistically bias the abstract's >5% DSC claim, but it is test-set selection bias / missing an independent validation split, not a reduction of the prediction to its inputs. Under the stated circularity definition, the score is 2.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central result rests on a set of standard diffusion and FD assumptions plus several hand-chosen hyperparameters. Most importantly, the feature extractor is selected using test-set performance, which is a free choice that affects the reported gains. No new physical entities are introduced.

free parameters (8)
  • Feature-extractor selection per dataset = MedDINOv3 (liver CT); BiomedCLIP (BraTS)
    Encoder used for FD-loss was selected based on downstream segmentation test performance (Table 3), then reused for final reported results in Table 2.
  • Queue size N = 5,000
    Chosen for stable FD estimation; inherited from Yang et al. [24].
  • Gradient-tracked batch size M = 16
    Batch size for gradient propagation through DDIM sampling.
  • Non-gradient images per iteration = 32
    Extra samples added to queue without gradients to accelerate queue turnover.
  • Finetuning learning rate = 1e-7
    Hand-chosen; no schedule described.
  • Finetuning steps = 20,000
    Hand-chosen; convergence monitored informally.
  • DDIM sampling steps = 25
    Finite-step approximation of the generative process; gradients through full chain.
  • Tikhonov epsilon = 1e-6
    Used in KL/Bhattacharyya ablation to stabilize matrix inversion.
assumptions (6)
  • domain assumption Diffusion model with velocity parameterization and mask conditioning is a valid base generator.
    Adopts pretrained segmentation-guided diffusion models from [9] and assumes they are representative.
  • standard math Feature distributions are well-approximated by multivariate Gaussians.
    FD definition in Eq. 1 assumes Gaussian features; typical for FID but may fail for complex medical features.
  • domain assumption Pretrained encoders (InceptionV3, RadInceptionV3, BiomedCLIP, MedDINOv3, MM-DINOv2) provide feature spaces whose statistics correlate with clinically relevant tumor morphology.
    Central premise that matching FD in these spaces improves downstream segmentation.
  • domain assumption 25-step DDIM sampling is differentiable and its gradients are a faithful training signal for the FD loss.
    Gradients through the full sampling chain are used; no analysis of bias/variance.
  • domain assumption Fixed seeds and a five-model soft-voting ensemble adequately control variance for the reported comparisons.
    No error bars or per-seed results are reported.
  • domain assumption Queue-based estimation with N=5000 generated images and incremental updates approximates the true generated distribution well enough for FD minimization.
    Inherited from [24]; no validation of queue convergence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Medical Image Generative Models with Fr\'echet Distance Loss." pith.science (2026). https://pith.science/paper/D26KXACX

@misc{pith2026260713300,
  author       = {Pith},
  title        = {Pith review of: Improving Medical Image Generative Models with Fr\'echet Distance Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D26KXACX}},
  note         = {Machine review of arXiv:2607.13300}
}
abstract

Diffusion generative models have demonstrated immense potential for synthetic medical image generation. However, these models often struggle to capture complex morphological characteristics of heterogeneous tumors with irregular boundaries, limiting their utility for downstream clinical tasks such as segmentation. This limitation stems from the standard denoising objective: minimizing a per-pixel error, which smooths high-variance irregular structures characteristic of tumors. To address this, we propose finetuning these generative models with Fr\'echet Distance loss (FD-loss). FD-loss aligns the first and second order feature statistics of real and generated images in a pretrained encoder space, encouraging the generator to capture complex structural variations characteristic of heterogeneous tumors. We integrate FD-loss across diverse architectural settings, using both natural- and medical-image encoders on multiple liver and brain cancer datasets spanning CT and MRI modalities. Downstream segmentation networks trained on our FD-regularized synthetic data consistently achieve superior performance, improving tumor DSC by $>$$5\%$ over unregularized synthetic augmentation alone. Qualitative analysis suggests these gains are associated with more faithful tumor synthesis and fewer segmentation hallucinations. Our results show FD-loss as an effective regularizer for medical image generative models to improve clinical workflows.

Figures

Figures reproduced from arXiv: 2607.13300 by the authors.

Figure 1
Figure 1. Our proposed segmentation-guided generative model finetuning with FD-loss. using 25-step DDIM sampling. These images are passed through the frozen ex￾tractor to obtain N synthetic-image features, which initialize a feature queue F of fixed size N used to estimate the generated-image distribution. During finetuning, each iteration samples a mini-batch of M = 16 segmen￾tation masks and synthesizes M images through a d… view at source ↗
Figure 2
Figure 2. Qualitative evaluation of FD-loss finetuning for image generation and down￾stream segmentation. Top: segmentation-conditioned synthetic images generated with different encoder spaces. Bottom: downstream liver CT segmentation results. validation loss. To mitigate the run-to-run variance inherent to single-seed U-Net training, we trained an ensemble of five models for each configuration. We used a fixed set of random … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 3 canonical work pages

  1. [1]

    Scientific Data4, 170117 (2017).https://doi.org/10.1038/sdata.2017.117

    Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., et al.: Advancing the cancer genome atlas glioma MRI col- lections with expert segmentation labels and radiomic features. Scientific Data4, 170117 (2017).https://doi.org/10.1038/sdata.2017.117

  2. [2]

    Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R.T., Berger, C., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge (2019).https://doi.org/10.48550/arXiv.1811.02629

  3. [3]

    Medical Image Analysis84, 102680 (2023).https://doi.org/10.1016/j.media.2022.102680

    Bilic,P.,Christ,P.,Li,H.B.,Vorontsov,E.,Ben-Cohen,A.,Kaissis,G.,Szeskin,A., Jacobs, C., et al.: The liver tumor segmentation benchmark (LiTS). Medical Image Analysis84, 102680 (2023).https://doi.org/10.1016/j.media.2022.102680

  4. [4]

    Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying MMD GANs (2021).https://doi.org/10.48550/arXiv.1801.01401

  5. [5]

    Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., et al.: MONAI: An open-source framework for deep learning in healthcare (2022).https://doi.org/10.48550/arXiv.2211.02701

  6. [6]

    https://doi.org/10.48550/arXiv.1706.08500

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium (2018). https://doi.org/10.48550/arXiv.1706.08500

  7. [7]

    In: Advances in Neural Information Processing Systems

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems. vol. 33, pp. 6840– 6851 (2020),https://proceedings.neurips.cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf

  8. [8]

    In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Jayasumana,S.,Ramalingam,S.,Veit,A.,Glasner,D.,Chakrabarti,A.,Kumar,S.: Rethinking FID: Towards a better evaluation metric for image generation. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9307–9315. IEEE (2024).https://doi.org/10.1109/CVPR52733.2024.00889

Show all 27 references
  1. [9]

    Marshall et al

    Konz, N., Chen, Y., Dong, H., Mazurowski, M.A.: Anatomically-controllable med- ical image generation with segmentation-guided diffusion models (2024).https: //doi.org/10.48550/arXiv.2402.05210 10 A. Marshall et al

  2. [10]

    Medical Image Analysis110, 103943 (2026)

    Konz, N., Osuala, R., Verma, P., Chen, Y., Gu, H., Dong, H., Chen, Y., Mar- shall, A., et al.: Fréchet radiomic distance (FRD): A versatile metric for com- paring medical imaging datasets. Medical Image Analysis110, 103943 (2026). https://doi.org/10.1016/j.media.2026.103943

  3. [11]

    Li, Y., Wu, Y., Lai, Y., Hu, M., Yang, X.: MedDINOv3: How to adapt vision foundation models for medical image segmentation? (2025).https://doi.org/ 10.48550/arXiv.2509.02379

  4. [12]

    https://doi.org/10.48550/arXiv.2504.06897

    Mao, J., Wang, Y., Tang, Y., Xu, D., Wang, K., Yang, Y., Zhou, Z., Zhou, Y.: MedSegFactory: Text-guided generation of medical image-mask pairs (2025). https://doi.org/10.48550/arXiv.2504.06897

  5. [13]

    Mathiasen, A., Hvilshøj, F.: Backpropagating through fréchet inception distance (2021).https://doi.org/10.48550/arXiv.2009.14075

  6. [14]

    Radiology: Artificial Intelligence4(5), e210315 (2022).https://doi.org/10.1148/ryai.210315

    Mei, X., Liu, Z., Robson, P.M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K.E., et al.: RadImageNet: An open radiologic deep learning re- search dataset for effective transfer learning. Radiology: Artificial Intelligence4(5), e210315 (2022).https://doi.or...

  7. [15]

    IEEE transactions on medical imaging34(10), 1993–2024 (2015).https://doi.org/10.1109/TMI.2014.2377694

    Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging34(10), 1993–2024 (2015).https://doi.org/10.1109/TMI.2014.2377694

  8. [16]

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation (2015).https://doi.org/10.48550/arXiv.1505.04597

  9. [17]

    1606.03498

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Im- proved techniques for training GANs (2016).https://doi.org/10.48550/arXiv. 1606.03498

  10. [18]

    Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models (2022).https://doi.org/10.48550/arXiv.2202.00512

  11. [19]

    Scholz, D., Erdur, A.C., Ehm, V., Meyer-Baese, A., Peeken, J.C., Rueckert, D., Wiestler, B.: MM-DINOv2: Adapting foundation models for multi-modal medical image analysis (2025).https://doi.org/10.48550/arXiv.2509.06617

  12. [20]

    Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models (2022).https: //doi.org/10.48550/arXiv.2010.02502

  13. [21]

    In: 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR)

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the in- ception architecture for computer vision. In: 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR). pp. 2818–2826. IEEE (2016). https://doi.org/10.1109/CVPR.2016.308

  14. [22]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI

    Woodland, M., Castelo, A., Al Taie, M., Albuquerque Marques Silva, J., Elta- her, M., Mohn, F., Shieh, A., Kundu, S., et al.: Feature extraction for gen- erative medical imaging evaluation: New evidence against an evolving trend. In: Medical Image Computing and Computer Assist...

  15. [23]

    Sci- entific Data13(1), 31 (2025).https://doi.org/10.1038/s41597-025-06343-4

    Wu, X., Su, H., Hua, Y., Xu, Y., Wang, L., Wang, X., Wang, S., Jin, B., et al.: A multi-phase CT dataset for automated differential diagnosis of liver tumors. Sci- entific Data13(1), 31 (2025).https://doi.org/10.1038/s41597-025-06343-4

  16. [24]

    Yang, J., Geng, Z., Ju, X., Tian, Y., Wang, Y.: Representation fréchet loss for visual generation.https://doi.org/10.48550/arXiv.2604.28190

  17. [25]

    Nature Communications16(1), 6486 (2025).https://doi.org/10.1038/ s41467-025-61754-6 Improving Medical Image Generative Models with Fréchet Distance Loss 11

    Zhang, L., Jindal, B., Alaa, A., Weinreb, R., Wilson, D., Segal, E., Zou, J., Xie, P.: Generative AI enables medical image segmentation in ultra low-data regimes. Nature Communications16(1), 6486 (2025).https://doi.org/10.1038/ s41467-025-61754-6 Improving Medical Image Genera...

  18. [26]

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., et al.: BiomedCLIP: a multimodal biomedical foundation model pre- trained from fifteen million scientific image-text pairs (2025).https://doi.org/ 10.48550/arXiv.2303.00915

  19. [2024]

    pp. 87–97. Springer Nature Switzerland (2024).https://doi.org/10.1007/ 978-3-031-72390-2_9

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.