{"id":"75014ac7-6dc7-48ef-91c9-88a6d6dc93e1","arxiv_id":"2411.15584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FLD+ evaluates generative models by exponentiating the ratio of average log-likelihoods of generated versus real images computed by a normalizing flow on pooled pretrained features.","lead":"This paper introduces FLD+, a metric for judging generated images by training a normalizing flow on features of real images and measuring how likely generated images are under that flow. The authors report that FLD+ needs only a few hundred images to stabilize, tracks image degradation monotonically, and can be retrained on new domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Monotonicity is demonstrated only on degradation self-similarity, not on the metric's claimed ordering power; the key comparison curves lack FID baselines and error bars, so the central advantage over FID is under-supported.","rationale":"I read the paper in good faith. The core proposal—fit a normalizing flow to pooled penultimate-layer features of a frozen pretrained backbone, then use the exponentiated ratio of average log-likelihoods as a quality metric—is coherent and the sample-efficiency argument is plausible because a flow gives a scalar log-likelihood per sample rather than a covariance estimate. The reader's weakest_assumption targets exactly the same load-bearing condition I identified: the sufficiency of the pooled ResNet-18 feature space for capturing quality-relevant, domain-specific differences. My independent check of the full text confirms the evidence gaps: the monotonicity figures (3, 4, 5, 6, 10) plot only FLD+ with no FID comparison and no error bars; the abstract promises occlusion and medical-image results that never appear; Table 1 has two rows with no human or FID baseline; and the sample-efficiency plot (Figure 7) reports FID stability without showing the standard deviations that the text says are in the appendix. The architecture choice is also a genuine risk: 2D average pooling of an 8x8x512 tensor to 4x4x512 and flattening to 8192 dimensions discards spatial fine structure, which is precisely where generative artifacts live, and a flow trained only on real features will assign high likelihood only to the training domain's feature manifold. None of this makes the method unsound; the metric is a definition, and the empirical claims are testable. But as published, the strongest claim is conditional: the monotonicity evidence is narrow, the sample-size stability is plausible but not rigorously quantified against FID at matched sample sizes, and the cross-domain adaptability assertion is untested. I therefore keep the CONDITIONAL verdict. The test I propose would settle the concern by checking monotonicity and ranking agreement on a held-out corruption type and against human judgments, which is the minimal evidence needed to support a drop-in replacement claim.","tokens_in":9904,"tokens_out":1964,"duration_ms":16112,"concrete_test":"Re-run the Figure 3-6 and Figure 10 experiments with the published code, adding (a) FID computed on the same image sets at the same sample sizes, (b) error bars over at least 5 independent flow trainings and real-image subsets, and (c) an additional corruption type not used in design, e.g., JPEG compression or inpainting artifacts. Then check whether FLD+ remains strictly monotonic and whether the SDXL-vs-SD1.5 ranking in Table 1 agrees with FID and with a human preference study on the same 1,000 prompts. If FLD+ fails monotonicity on any added corruption or reverses the model ranking, the central claim is falsified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that FLD+ is strongly monotonic with respect to noise, occlusion, diffusion steps, and model size, and that it is stable with two orders of magnitude fewer images than FID. The load-bearing assumption is that the ratio of average log-likelihoods of generated vs real images, computed in a heavily downsampled 8192-dim average-pooled ResNet-18 feature space (Eqs. 1-3, Eq. 6), preserves the fine-grained and domain-specific information needed to order generative models by perceptual quality. The paper never directly validates this ordering against a reference metric or human ratings: Figures 3-6 and 10 show only FLD+ curves without FID baselines, without error bars on the monotonicity plots, and without any external ground truth that the degradation ordering is the correct perceptual ordering. The claimed medical-domain adaptability (Abstract, Section 3.3) is asserted but never run. More importantly, the metric is a ratio of average log-likelihoods of a flow trained on real data, so any degradation that systematically moves features toward the high-density region of the real training distribution could produce non-monotonic or inverted behavior; the paper shows monotonicity only for a few hand-picked distortions, and Table 1 compares only two model sizes with no statistical test beyond std error. The occlusion experiment promised in the abstract is absent from the full text. These omissions mean the central claim of drop-in superiority over FID rests on unverified feature-space sufficiency and incomplete comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLD+ (Flow-based Likelihood Distance Plus), an evaluation metric for generative models that computes the exponentiated ratio of average log-likelihoods of generated versus real images under a normalizing flow trained on features from a frozen, ImageNet-pretrained ResNet-18 backbone with 2D average pooling (Eqs. 1-6). The authors claim that FLD+ is strongly monotonic with respect to image degradations (noise, blur, salt-and-pepper, diffusion steps, and model size), stable with two orders of magnitude fewer images than FID, and easily adaptable to new domains such as medical images. Experiments on CelebA-HQ include monotonicity sweeps for Gaussian noise, Gaussian blur, salt-and-pepper noise, DDPM diffusion steps, a two-model comparison (Stable Diffusion v1.5 vs. SDXL), and a sample-efficiency comparison with FID. The paper also includes an ablation of backbones and pooling, a toy experiment showing FLD+ captures deviations invisible to FID under non-Gaussian mixtures, and a computational efficiency comparison against FLD.","tokens_in":10154,"tokens_out":7827,"duration_ms":66184,"significance":"If the claims were fully substantiated, FLD+ would be a valuable drop-in replacement for FID in data-scarce regimes and for new domains. The metric definition is clear and the idea of fitting a normalizing flow to lower-dimensional features is sensible. The paper provides a reproducibility-friendly code link and a useful efficiency analysis (12x faster training and 6x faster evaluation than FLD). However, the current evidence is insufficient for the central claims: occlusion and medical-domain experiments promised in the abstract are absent, monotonicity is not compared against FID on the same distortion sweeps, several plots lack error bars, and the model-size monotonicity rests on only two data points. The sample-efficiency claim also needs stronger FID baseline evidence. With additional experiments and more careful claims, the metric could be a meaningful contribution.","major_comments":[{"comment":"The abstract lists occlusion as one of the degradations for which FLD+ is strongly monotonic, but no occlusion experiment appears anywhere in the manuscript. The distortion experiments cover Gaussian noise, Gaussian blur, and salt-and-pepper noise only. Please add an occlusion sweep (e.g., random rectangular masks with increasing coverage) or revise the abstract to list only the degradations actually tested.","section":"Abstract and Section 4.2"},{"comment":"The monotonicity plots are shown as single curves without error bars or multiple random seeds, and FID is not computed on the same distortion sweeps. The phrase 'unlike FID' (Abstract) is therefore not directly supported in this paper. Add FID curves on the identical sweeps and report mean ± standard deviation over at least five seeds so that the monotonicity claim is statistically grounded.","section":"Figures 3, 4, 5, and 6"},{"comment":"The claim of 'strongly monotonic behavior with respect to generative model size' is based on only two model sizes (Stable Diffusion v1.5 at 0.9B and SDXL at 3.5B). A two-point comparison cannot establish monotonicity; any two points are trivially ordered. Add at least one additional model size (e.g., SD 2.1 or a distilled variant) or restate the claim as 'lower FLD+ for the larger model in this pairwise comparison.'","section":"Table 1"},{"comment":"The abstract and Section 3.3 claim that FLD+ can be easily retrained on new domains such as medical images, but no medical-image experiment is presented. The phrase 'We also show' in the abstract is unsupported. Add an experiment in which the flow is retrained on a medical imaging dataset (e.g., CheXpert or BraTS) and FLD+ is used to rank models, or remove the medical-domain claim.","section":"Abstract and Section 3.3"},{"comment":"The sample-efficiency claim of 'two orders of magnitude fewer images' is under-supported. The figure shows FLD+ and FID up to 10,000 samples, but the text states FID requires more than 20,000 samples; this point is outside the plotted range, and no FID error bars are shown. Provide FID stability curves (mean ± std) for sample sizes including 10,000, 20,000, and 50,000, and define a quantitative stability criterion (e.g., within 1% of the asymptotic value) to substantiate the claim.","section":"Section 4.4 and Figure 7"},{"comment":"The metric value depends on the trained flow, but the manuscript gives no training details for the neural spline flow (number of epochs, learning rate, batch size, number of flow layers, or the number of real images used for training). Since likelihood estimates are sensitive to flow optimization, report these hyperparameters and provide a sensitivity analysis (e.g., train the flow with different random seeds or learning rates) to support the 'stable' and 'reliable' claims.","section":"Section 3.2 and Eq. (6)"}],"minor_comments":[{"comment":"The text contains a typo: 'V AEs' should be 'VAEs' in Section 3.1.","section":"Section 3.1"},{"comment":"The y-axis label 'FLD+' and the title '# Sampes' contain typos; 'Sampes' should be 'Samples'.","section":"Figure 7"},{"comment":"The sentence 'Figure 10 and 5 illustrate the behavior of FLD+ when Gaussian blur and salt-and-pepper noise are applied' incorrectly references Figure 10, which is the StyleGAN2 figure in the Appendix. The correct references are Figure 4 (Gaussian blur) and Figure 5 (salt-and-pepper noise).","section":"Section 4.2"},{"comment":"The noise parameters are given as α ∈ {0, 0.001, 0.005, 0.01, etc.}, but Figure 3 shows α values such as 0.005, 0.02, 0.2, and 0.5. Please specify the full set of values used in each sweep so the experiments are reproducible.","section":"Appendix A"},{"comment":"The sentence 'We made FLD+ computationally even more efficient by applying normalizing flows to features extracted in a lower-dimensional latent space instead of using a pre-trained network' is misleading because the method does use a pre-trained network. Rephrase to 'instead of operating directly on the image space.'","section":"Section 3.2"},{"comment":"The symbol L_r is used both for the log-likelihood of real images (Eq. 4) and for the log-likelihood of generated images in Eq. (6), which is confusing. Define a single symbol, e.g., ℓ(x) = N(f(x), θ), for the log-likelihood under the real-data flow.","section":"Equations (4) and (6)"},{"comment":"The caption says 'The leftmost image represents a reference mixture-of-Gaussian distribution,' but no images are shown in the table. Describe how the mixture-of-Gaussians toy images are generated and how FLD+ features are computed in this example so that the experiment is reproducible.","section":"Table 2"},{"comment":"Reference [1] has inconsistent year and venue formatting: it lists '2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9307–9315, 2023.' Please correct the year/venue.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an early preprint with a promising idea but with a notable mismatch between the abstract's claims and the experiments. The missing occlusion and medical-domain experiments, the lack of FID baselines on the monotonicity sweeps, and the two-point model-size comparison are all fixable with additional experiments. The code release is a strength. The relationship to the authors' prior FLD work [3] should also be clarified; the novelty relative to FLD is mostly in the feature-space flow and the efficiency gains, not in the likelihood-ratio principle. I recommend major revision rather than rejection, provided the authors can supply the missing evidence or soften the claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FLD+ is a legitimate incremental extension of the authors' own FLD — replace pixel-space flow with a flow on average-pooled ResNet-18 features and take the exponentiated ratio of average log-likelihoods. The definition is clear and the empirical core is mostly honest. The paper overclaims in the abstract and leaves out promised experiments; that is the main problem.\n\nWhat's new: the specific metric formulation, the lower-dimensional feature-space variant, and the sample-efficiency comparison with FID. The authors also report concrete numbers — 12x faster training than FLD, roughly 300x faster than FID at 20k samples — and they released code. The normality-violation example (Table 2) is a nice sanity check that the likelihood ratio sees distributional differences FID's Gaussian assumption misses. The sample-efficiency figure does support the two-orders-of-magnitude claim, and standard deviations are in the appendix.\n\nWhere it gets soft. First, the abstract promises monotonicity for occlusion and medical-image adaptability; neither appears in the experiments. That is a mismatch between the headline and the actual content. Second, the monotonicity plots (Figures 3-6, 10) show FLD+ curves without FID baselines on the same distortion sweeps and without error bars in the main text. The claim \"unlike FID\" is borrowed from reference [1] rather than demonstrated here. Third, Table 1 compares two model sizes with no test beyond standard error; calling that \"monotonic with generative model size\" is thin. Fourth, the metric's discriminating power depends on the averaged pooled features preserving fine-grained artifacts; the paper does not test that outside CelebA-HQ-style photographs. That is a real limitation, not a fatal one — the method is plausible and could be validated with a few extra experiments.\n\nThe citation pattern is fine: FLD+ builds on the authors' FLD [3], and the self-citation is appropriate. No circular fitting of constants; the metric is a definition.\n\nWho this is for: people building or benchmarking generative-model evaluation metrics. It deserves a serious referee, with the expectation of heavy revision: add FID baselines on identical distortion sweeps, add error bars to the monotonicity plots, run or remove the occlusion and medical claims, and widen the model-size comparison. I'd send it out.","headline":"A credible but overclaimed incremental metric: the flow-likelihood ratio idea is sound and the sample-efficiency evidence is real, but the paper promises occlusion and medical-domain tests it never runs.","tokens_in":10697,"tokens_out":2385,"would_cite":true,"duration_ms":22336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLD+ is a normalizing-flow metric that scores generative image models by the likelihood ratio between generated and real features, achieving stable results with roughly 300 images and monotone sensitivity to noise, blur, diffusion step…","keywords":["evaluation metric","generative models","normalizing flows","FID","likelihood ratio","image quality","sample efficiency","monotonicity"],"falsifier":"Construct two image sets that are perceptually or semantically distinct but produce identical average-pooled ResNet-18 feature distributions (e.g., by swapping textures or hiding artifacts in low-frequency components); if FLD+ returns the same score while human raters or downstream metrics see a clear quality gap, the metric is blind to the features that matter. A second test is to run the claimed medical-image adaptation on a public medical dataset and check whether FLD+ increases monotonically with clinically defined degradation (e.g., compression or motion blur) and whether its model ranking matches human expert ratings.","tokens_in":9654,"feed_emoji":"🖼️","tokens_out":5203,"duration_ms":42924,"temperature":0.7,"pith_summary":"The paper proposes FLD+, a metric for judging how well a generative image model matches a real image distribution. Instead of assuming the feature vectors of real and generated images are Gaussian, as Fréchet Inception Distance (FID) does, FLD+ fits a normalizing flow to pooled features from a frozen ImageNet-pretrained ResNet-18 and measures the ratio of average log-likelihoods between generated and real images under that flow. The authors claim this ratio rises monotonically as images degrade—with Gaussian noise, blur, salt-and-pepper noise, earlier diffusion steps, and smaller generative models—and that a stable estimate needs only a few hundred images rather than the 20,000-plus FID requires. If true, FLD+ would give researchers a cheap, domain-adaptable way to compare generative models and to monitor quality during training.","feed_headline":"Image-quality metric beats FID with 100x fewer images","feed_subtitle":"Normalizing flows give stable, monotonic quality scores from a few hundred images, where FID needs 20,000.","key_machinery":"The central object is the ratio of average log-likelihoods between generated and real images under a normalizing flow. A frozen ImageNet-pretrained ResNet-18 backbone extracts a feature tensor, a 2D average-pooling layer reduces it (e.g., from $8\\times8\\times512$ to $4\\times4\\times512$), and a neural spline flow models the distribution of the flattened 8192-dimensional vectors. Because the flow computes exact log-likelihoods via the change-of-variables formula, the metric does not require the Gaussian assumption that FID relies on; the exponentiated ratio then converts a likelihood deficit into a positive score that grows as the generated distribution drifts from the real one.","core_discovery":"The central claim is that a normalizing flow fitted to low-dimensional, average-pooled features of a pre-trained backbone yields a generative-model evaluation metric that is monotone with respect to image degradation, stable at two orders of magnitude fewer samples than FID, and cheap to retrain on new domains. Concretely, FLD+ is computed as $\\exp\\left(\\frac{\\sum_{x\\in G} L_r(x)/|G|}{\\sum_{x\\in R} L_r(x)/|R|}\\right)$, where $L_r$ is the log-likelihood assigned by a rational-quadratic neural spline flow trained on real images' features. The paper demonstrates monotonic increases for Gaussian noise, blur, salt-and-pepper noise, earlier steps in a DDPM diffusion process, and when comparing a smaller Stable Diffusion v1.5 model against the larger SDXL, and it contrasts these results with FID's known non-monotonicity.","pith_inferences":["The likelihood ratio formulation implies FLD+ is not symmetric and does not satisfy triangle-inequality-like properties; it is a directed similarity score, and its absolute scale depends on the flow model and feature extractor, so cross-paper comparisons would need a standardized calibration or a reference set.","A natural stress test beyond the paper is to apply FLD+ to a domain where the ImageNet-pretrained backbone is known to be weak, such as X-ray or histopathology images, and compare FLD+ ordering against human perceptual ratings or downstream task performance.","If the monotonicity holds for arbitrary degradation operators, FLD+ could be adapted as a perceptual quality index by computing the likelihood ratio against a clean reference set, potentially competing with full-reference metrics like LPIPS.","The strong dependence on the pooling operation (average vs max) suggests that the choice of feature aggregation is a hyperparameter that could be tuned per domain; the paper only tests two pooling choices, so a broader search might further improve sensitivity."],"forward_implications":["FLD+ can replace FID in settings with limited real or generated data, such as medical imaging, where collecting tens of thousands of images is impractical.","The monotonic behavior with diffusion steps suggests FLD+ could serve as a validation signal during diffusion model training or sampling, flagging when early steps are over-weighted.","The small-sample stability (under ~300 images) makes it feasible to embed FLD+ inside training loops for real-time model selection, something FID cannot do without large batches.","Because the flow is retrainable on a new domain with a few thousand images, FLD+ offers a path to domain-specific evaluation without retraining a large backbone."],"supporting_citations":[{"why":"Documents FID's non-monotonic behavior with degradation and the normality-assumption failure that FLD+ aims to fix.","marker":"[1]"},{"why":"Defines FID, the baseline metric that FLD+ is compared against throughout the paper.","marker":"[2]"},{"why":"Introduces the predecessor FLD metric, which FLD+ improves by operating on lower-dimensional features.","marker":"[3]"},{"why":"Provides the normalizing-flow theory that allows exact log-likelihood computation, the core mechanism of FLD+.","marker":"[14]"},{"why":"Supplies the rational-quadratic neural spline flow architecture used as the normalizing flow in FLD+.","marker":"[16]"},{"why":"Defines the ResNet-18 backbone used as the frozen feature extractor in FLD+.","marker":"[17]"},{"why":"Provides the DDPM model used to generate progressive diffusion steps for monotonicity evaluation.","marker":"[18]"},{"why":"The larger Stable Diffusion XL model whose output is rated higher by FLD+ than the smaller model.","marker":"[20]"},{"why":"The smaller Stable Diffusion v1.5 model used as the lower-quality comparison in the model-size experiment.","marker":"[21]"}],"fun_headline_variants":["FLD+ evaluates image models with 100x fewer images","Flow-based metric FLD+ stays stable on tiny datasets","Monotonic image quality metric needs 100x less data than FID","FLD+: data-efficient, retrainable, monotonic evaluation metric","New generative-model metric outperforms FID at 100x smaller sample size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's usefulness rests on the assumption that average-pooled features from an ImageNet-pretrained ResNet-18 retain the fine-grained information needed to tell real from generated images, and that a normalizing flow fitted to those features yields likelihood ratios that order models by quality in any target domain.","fun_headline_variants_meta":{"raw":{"variants":["FLD+ evaluates image models with 100x fewer images","Flow-based metric FLD+ stays stable on tiny datasets","Monotonic image quality metric needs 100x less data than FID","FLD+: data-efficient, retrainable, monotonic evaluation metric","New generative-model metric outperforms FID at 100x smaller sample size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2969,"prompt_tokens":956,"completion_tokens":2013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1922}},"tokens_in":572,"tokens_out":2013,"duration_ms":12525,"temperature":1.0,"reasoning_tokens":1922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:07:46.687664+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two image sets that are perceptually or semantically distinct but produce identical average-pooled ResNet-18 feature distributions (e.g., by swapping textures or hiding artifacts in low-frequency components); if FLD+ returns the same score while human raters or downstream metrics see a clear quality gap, the metric is blind to the features that matter. A second test is to run the claimed medical-image adaptation on a public medical dataset and check whether FLD+ increases monotonically with clinically defined degradation (e.g., compression or motion blur) and whether its model ranking matches human expert ratings.","supporting_citations":[{"cited_title":"Rethinking fid: Towards a better evaluation metric for image generation.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9307–9315, 2023","cited_arxiv_id":null,"evidence_quote":"Documents FID's non-monotonic behavior with degradation and the normality-assumption failure that FLD+ aims to fix."},{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","cited_arxiv_id":null,"evidence_quote":"Defines FID, the baseline metric that FLD+ is compared against throughout the paper."},{"cited_title":"Normalizing flow-based metric for image generation, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the predecessor FLD metric, which FLD+ improves by operating on lower-dimensional features."},{"cited_title":"Prince, and Marcus A","cited_arxiv_id":null,"evidence_quote":"Provides the normalizing-flow theory that allows exact log-likelihood computation, the core mechanism of FLD+."}],"review_version":1}