REVIEW 6 major objections 8 minor 1 cited by
FLD+: Data-efficient Evaluation Metric for Generative Models
T0 review · 6 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read FLD+ is a normalizing-flow metric that scores generative image models by the likelihood ratio between generated and real features, achieving stable results with roughly 300 images and monotone sensitivity to noise, blur, diffusion step…
desk verdict A credible but overclaimed incremental metric: the flow-likelihood ratio idea is sound and the sample-efficiency evidence is real, but the paper promises occlusion and medical-domain tests it never runs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ratio of average log-likelihoods between generated and real images under a normalizing flow. A frozen ImageNet-pretrained ResNet-18 backbone extracts a feature tensor, a 2D average-pooling layer reduces it (e.g., from $8\times8\times512$ to $4\times4\times512$), and a neural spline flow models the distribution of the flattened 8192-dimensional vectors. Because the flow computes exact log-likelihoods via the change-of-variables formula, the metric does not require the Gaussian assumption that FID relies on; the exponentiated ratio then converts a likelihood deficit into a positive score that grows as the generated distribution drifts from the real one.
What would settle it
Construct two image sets that are perceptually or semantically distinct but produce identical average-pooled ResNet-18 feature distributions (e.g., by swapping textures or hiding artifacts in low-frequency components); if FLD+ returns the same score while human raters or downstream metrics see a clear quality gap, the metric is blind to the features that matter. A second test is to run the claimed medical-image adaptation on a public medical dataset and check whether FLD+ increases monotonically with clinically defined degradation (e.g., compression or motion blur) and whether its model ranking matches human expert ratings.
Extended reading notes
Core claim
The central claim is that a normalizing flow fitted to low-dimensional, average-pooled features of a pre-trained backbone yields a generative-model evaluation metric that is monotone with respect to image degradation, stable at two orders of magnitude fewer samples than FID, and cheap to retrain on new domains. Concretely, FLD+ is computed as $\exp\left(\frac{\sum_{x\in G} L_r(x)/|G|}{\sum_{x\in R} L_r(x)/|R|}\right)$, where $L_r$ is the log-likelihood assigned by a rational-quadratic neural spline flow trained on real images' features. The paper demonstrates monotonic increases for Gaussian noise, blur, salt-and-pepper noise, earlier steps in a DDPM diffusion process, and when comparing a smaller Stable Diffusion v1.5 model against the larger SDXL, and it contrasts these results with FID's known non-monotonicity.
Load-bearing premise
The metric's usefulness rests on the assumption that average-pooled features from an ImageNet-pretrained ResNet-18 retain the fine-grained information needed to tell real from generated images, and that a normalizing flow fitted to those features yields likelihood ratios that order models by quality in any target domain.
Editorial extensions
If this is right
- FLD+ can replace FID in settings with limited real or generated data, such as medical imaging, where collecting tens of thousands of images is impractical.
- The monotonic behavior with diffusion steps suggests FLD+ could serve as a validation signal during diffusion model training or sampling, flagging when early steps are over-weighted.
- The small-sample stability (under ~300 images) makes it feasible to embed FLD+ inside training loops for real-time model selection, something FID cannot do without large batches.
- Because the flow is retrainable on a new domain with a few thousand images, FLD+ offers a path to domain-specific evaluation without retraining a large backbone.
Reading between the lines
- The likelihood ratio formulation implies FLD+ is not symmetric and does not satisfy triangle-inequality-like properties; it is a directed similarity score, and its absolute scale depends on the flow model and feature extractor, so cross-paper comparisons would need a standardized calibration or a reference set.
- A natural stress test beyond the paper is to apply FLD+ to a domain where the ImageNet-pretrained backbone is known to be weak, such as X-ray or histopathology images, and compare FLD+ ordering against human perceptual ratings or downstream task performance.
- If the monotonicity holds for arbitrary degradation operators, FLD+ could be adapted as a perceptual quality index by computing the likelihood ratio against a clean reference set, potentially competing with full-reference metrics like LPIPS.
- The strong dependence on the pooling operation (average vs max) suggests that the choice of feature aggregation is a hyperparameter that could be tuned per domain; the paper only tests two pooling choices, so a broader search might further improve sensitivity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FLD+ (Flow-based Likelihood Distance Plus), an evaluation metric for generative models that computes the exponentiated ratio of average log-likelihoods of generated versus real images under a normalizing flow trained on features from a frozen, ImageNet-pretrained ResNet-18 backbone with 2D average pooling (Eqs. 1-6). The authors claim that FLD+ is strongly monotonic with respect to image degradations (noise, blur, salt-and-pepper, diffusion steps, and model size), stable with two orders of magnitude fewer images than FID, and easily adaptable to new domains such as medical images. Experiments on CelebA-HQ include monotonicity sweeps for Gaussian noise, Gaussian blur, salt-and-pepper noise, DDPM diffusion steps, a two-model comparison (Stable Diffusion v1.5 vs. SDXL), and a sample-efficiency comparison with FID. The paper also includes an ablation of backbones and pooling, a toy experiment showing FLD+ captures deviations invisible to FID under non-Gaussian mixtures, and a computational efficiency comparison against FLD.
Significance. If the claims were fully substantiated, FLD+ would be a valuable drop-in replacement for FID in data-scarce regimes and for new domains. The metric definition is clear and the idea of fitting a normalizing flow to lower-dimensional features is sensible. The paper provides a reproducibility-friendly code link and a useful efficiency analysis (12x faster training and 6x faster evaluation than FLD). However, the current evidence is insufficient for the central claims: occlusion and medical-domain experiments promised in the abstract are absent, monotonicity is not compared against FID on the same distortion sweeps, several plots lack error bars, and the model-size monotonicity rests on only two data points. The sample-efficiency claim also needs stronger FID baseline evidence. With additional experiments and more careful claims, the metric could be a meaningful contribution.
major comments (6)
- [Abstract and Section 4.2] The abstract lists occlusion as one of the degradations for which FLD+ is strongly monotonic, but no occlusion experiment appears anywhere in the manuscript. The distortion experiments cover Gaussian noise, Gaussian blur, and salt-and-pepper noise only. Please add an occlusion sweep (e.g., random rectangular masks with increasing coverage) or revise the abstract to list only the degradations actually tested.
- [Figures 3, 4, 5, and 6] The monotonicity plots are shown as single curves without error bars or multiple random seeds, and FID is not computed on the same distortion sweeps. The phrase 'unlike FID' (Abstract) is therefore not directly supported in this paper. Add FID curves on the identical sweeps and report mean ± standard deviation over at least five seeds so that the monotonicity claim is statistically grounded.
- [Table 1] The claim of 'strongly monotonic behavior with respect to generative model size' is based on only two model sizes (Stable Diffusion v1.5 at 0.9B and SDXL at 3.5B). A two-point comparison cannot establish monotonicity; any two points are trivially ordered. Add at least one additional model size (e.g., SD 2.1 or a distilled variant) or restate the claim as 'lower FLD+ for the larger model in this pairwise comparison.'
- [Abstract and Section 3.3] The abstract and Section 3.3 claim that FLD+ can be easily retrained on new domains such as medical images, but no medical-image experiment is presented. The phrase 'We also show' in the abstract is unsupported. Add an experiment in which the flow is retrained on a medical imaging dataset (e.g., CheXpert or BraTS) and FLD+ is used to rank models, or remove the medical-domain claim.
- [Section 4.4 and Figure 7] The sample-efficiency claim of 'two orders of magnitude fewer images' is under-supported. The figure shows FLD+ and FID up to 10,000 samples, but the text states FID requires more than 20,000 samples; this point is outside the plotted range, and no FID error bars are shown. Provide FID stability curves (mean ± std) for sample sizes including 10,000, 20,000, and 50,000, and define a quantitative stability criterion (e.g., within 1% of the asymptotic value) to substantiate the claim.
- [Section 3.2 and Eq. (6)] The metric value depends on the trained flow, but the manuscript gives no training details for the neural spline flow (number of epochs, learning rate, batch size, number of flow layers, or the number of real images used for training). Since likelihood estimates are sensitive to flow optimization, report these hyperparameters and provide a sensitivity analysis (e.g., train the flow with different random seeds or learning rates) to support the 'stable' and 'reliable' claims.
minor comments (8)
- [Section 3.1] The text contains a typo: 'V AEs' should be 'VAEs' in Section 3.1.
- [Figure 7] The y-axis label 'FLD+' and the title '# Sampes' contain typos; 'Sampes' should be 'Samples'.
- [Section 4.2] The sentence 'Figure 10 and 5 illustrate the behavior of FLD+ when Gaussian blur and salt-and-pepper noise are applied' incorrectly references Figure 10, which is the StyleGAN2 figure in the Appendix. The correct references are Figure 4 (Gaussian blur) and Figure 5 (salt-and-pepper noise).
- [Appendix A] The noise parameters are given as α ∈ {0, 0.001, 0.005, 0.01, etc.}, but Figure 3 shows α values such as 0.005, 0.02, 0.2, and 0.5. Please specify the full set of values used in each sweep so the experiments are reproducible.
- [Section 3.2] The sentence 'We made FLD+ computationally even more efficient by applying normalizing flows to features extracted in a lower-dimensional latent space instead of using a pre-trained network' is misleading because the method does use a pre-trained network. Rephrase to 'instead of operating directly on the image space.'
- [Equations (4) and (6)] The symbol L_r is used both for the log-likelihood of real images (Eq. 4) and for the log-likelihood of generated images in Eq. (6), which is confusing. Define a single symbol, e.g., ℓ(x) = N(f(x), θ), for the log-likelihood under the real-data flow.
- [Table 2] The caption says 'The leftmost image represents a reference mixture-of-Gaussian distribution,' but no images are shown in the table. Describe how the mixture-of-Gaussians toy images are generated and how FLD+ features are computed in this example so that the experiment is reproducible.
- [References] Reference [1] has inconsistent year and venue formatting: it lists '2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9307–9315, 2023.' Please correct the year/venue.
Circularity Check
No significant circularity; FLD+ is a likelihood-ratio definition with independent empirical support.
full rationale
No circular step is present. Equation (6) defines FLD+ as the exponentiated ratio of average log-likelihoods of generated and real images under a flow trained only on real data; no fitted parameter is renamed as a prediction, and no equation reduces to its own input by construction. The monotonicity, sample-efficiency, and model-comparison claims are empirical results (Figures 3-6, 7, Table 1) rather than consequences of the definition. Self-citations to the authors' prior FLD paper [3] appear as background and as a training-speed comparison (Section 4.4), but the load-bearing monotonicity experiments are independently rerun in this paper, so the argument does not reduce to a self-citation chain. The paper does have weaknesses that are not circularity: the medical-domain adaptation and occlusion experiments promised in the Abstract are not run in the full text, and the monotonicity plots lack FID baselines and error bars, making the comparison to FID under-supported. These are evidence-quality and completeness concerns, not circularity. Accordingly, the derivation is self-contained apart from a minor, non-load-bearing self-citation, yielding a score of 1.
Assumptions & free parameters
assumptions (4)
- standard math The change-of-variables formula gives exact log-likelihoods for normalizing flows.
- domain assumption Average-pooled ResNet-18 features preserve enough information to distinguish real from generated images and to respond monotonically to degradation.
- domain assumption The average log-likelihood ratio, after exponentiation, is a meaningful distance that orders generative models by quality.
- ad hoc to paper An ImageNet-pretrained backbone remains informative for new domains after only the flow is retrained.
Cite this review
Pith. "Pith review of FLD+: Data-efficient Evaluation Metric for Generative Models." pith.science (2026). https://pith.science/paper/PGKHVL4B
@misc{pith2026241115584,
author = {Pith},
title = {Pith review of: FLD+: Data-efficient Evaluation Metric for Generative Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PGKHVL4B}},
note = {Machine review of arXiv:2411.15584}
}
read the original abstract
We introduce a new metric to assess the quality of generated images that is more reliable, data-efficient, compute-efficient, and adaptable to new domains than the previous metrics, such as Fr\'echet Inception Distance (FID). The proposed metric is based on normalizing flows, which allows for the computation of density (exact log-likelihood) of images from any domain. Thus, unlike FID, the proposed Flow-based Likelihood Distance Plus (FLD+) metric exhibits strongly monotonic behavior with respect to different types of image degradations, including noise, occlusion, diffusion steps, and generative model size. Additionally, because normalizing flow can be trained stably and efficiently, FLD+ achieves stable results with two orders of magnitude fewer images than FID (which requires more images to reliably compute Fr\'echet distance between features of large samples of real and generated images). We made FLD+ computationally even more efficient by applying normalizing flows to features extracted in a lower-dimensional latent space instead of using a pre-trained network. We also show that FLD+ can easily be retrained on new domains, such as medical images, unlike the networks behind previous metrics -- such as InceptionNetV3 pre-trained on ImageNet.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Interpretable EEG-to-Image Generation with Semantic Prompts
A transformer-based EEG encoder retrieves multilevel LLM captions that condition a pretrained diffusion model, reporting high IS and KID on EEGCVPR, but without controls needed to attribute the gains to EEG decoding.
Reference graph
Works this paper leans on
-
[1]
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9307–9315, 2023
work page 2024
-
[3]
Normalizing flow-based metric for image generation, 2024
Pranav Jeevan, Neeraj Nixon, and Amit Sethi. Normalizing flow-based metric for image generation, 2024
work page 2024
-
[2]
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems, 2017
work page 2017
-
[4]
Improved techniques for training gans
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 2234–2242, Red Hook, NY , USA, 2016. Curran Associates Inc
work page 2016
-
[5]
Sutherland, Michael Arbel, and Arthur Gretton
Mikołaj Bi´nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations , 2018
work page 2018
-
[6]
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell., 43(12):4217–4228, December 2021
work page 2021
-
[7]
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014
2014
-
[8]
Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S
Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S. Bernstein. Hype: A benchmark for human eye perceptual evaluation of generative models, 2019
work page 2019
Show all 24 references
-
[9]
The fréchet distance between multivariate normal distributions
D.C Dowson and B.V Landau. The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):450–455, 1982
1982
-
[10]
Effectively unbiased fid and inception score and where to find them
Min Jin Chong and David Alexander Forsyth. Effectively unbiased fid and inception score and where to find them. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6069–6078, 2019. 10 Flow-based Likelihood Distance Plus
2020
-
[11]
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2015
2016
-
[12]
On aliased resizing and surprising subtleties in gan evaluation, 2022
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation, 2022
2022
-
[13]
Simon J.D. Prince. Understanding Deep Learning. The MIT Press, 2023
2023
-
[14]
Prince, and Marcus A
Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(11):3964–3979, November 2021
2021
-
[15]
Progressive growing of gans for improved quality, stability, and variation, 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, 2018
2018
-
[16]
Neural spline flows, 2019
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows, 2019
2019
-
[17]
Deep residual learning for image recognition, 2015
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015
2015
-
[18]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc
2020
-
[19]
Stable diffusion 100k custom prompts and images, 2023
Ryan Turley. Stable diffusion 100k custom prompts and images, 2023. Accessed: 2024-11-12
2023
-
[20]
Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023
2023
-
[21]
High-resolution image synthesis with latent diffusion models, 2022
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022
2022
-
[22]
Analyzing and improving the image quality of stylegan, 2020
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2020
2020
-
[23]
Le, and Hartwig Adam
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V . Le, and Hartwig Adam. Searching for mobilenetv3, 2019
2019
-
[24]
Ddpm: Pytorch implementation of denoising diffusion probabilistic models
Tianqi Chen. Ddpm: Pytorch implementation of denoising diffusion probabilistic models. https://github. com/tqch/ddpm-torch, 2023. Accessed: 2024-09-30. A Details of Distortions Gaussian Noise: We construct a noise matrix N with values drawn from a N (0, 1) Gaussian distributio...
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.