{"id":"87b4254e-b527-477d-b212-c17a10b8a1aa","arxiv_id":"2411.16171","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proposes a retrieval-based diversity metric (IRS), finds that state-of-the-art diffusion models retrieve at most 77% of training images, and introduces feature-conditioned DiADM to improve unconditional diversity.","lead":"Researchers introduce the Image Retrieval Score (IRS), which measures how many real training images a generative model can retrieve with its generated images, and use it to show that current diffusion models cover at most 77% of their training data. They also propose Diversity-Aware Diffusion Models (DiADM), which condition on image features instead of class labels to widen sampled diversity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adjusted IRS can exceed 1 (FFHQ 1.51, ChestX-ray14 1.08), so IRS∞,a is not a bounded measure of training-data diversity; the 77% ceiling and DiADM gains depend on this normalization.","rationale":"The reader's conditional verdict already identifies the feature-space nearest-neighbor assumption and the measurement-gap normalization as the weak load-bearing points. I agree with that assessment, but I would sharpen it: the single most decisive problem is that IRS∞,a is not anchored to a bounded notion of training diversity. The paper explicitly allows adjusted scores above 1 and then reports values of 1.51 and 1.08 in Table 3, which cannot be interpreted as fractions of training-data diversity under the paper's own definition. The normalization divides by IRS∞,real, which is itself a feature-extractor-dependent underestimate of real-data diversity, so the ratio encodes how differently the synthetic and real feature distributions collapse, not how much of the true data distribution is covered. The 77% ceiling claim is a direct casualty: it is computed after this normalization, so unless the normalization is validated, the ceiling is not a reliable statement about state-of-the-art models. The class-removal experiment in Fig. 5 is a useful sanity check and shows the metric can track a known diversity axis, but it uses manually removed classes and does not test the super-1 regime or cross-extractor stability. The DiADM 'without loss of image quality' claim is also entangled with this because Table 3 reports FID and IRS∞,a on the same DiADM rows; the inflated IRS∞,a values make it unclear whether diversity actually improved or the denominator shrank. I do not think this requires immediate rejection, because a focused re-analysis with alternative extractors and label-based coverage could resolve whether the normalization is valid. The reader's conditional verdict already captures that uncertainty, so I would leave the verdict unchanged rather than move it to accept or reject.","tokens_in":21637,"tokens_out":6691,"duration_ms":70896,"concrete_test":"Re-run the FFHQ and ChestX-ray14 DiADM evaluations from Table 3 with two feature extractors (SwAV and DINOv2) and report raw IRS∞,snth, IRS∞,real, and IRS∞,a. Then compare the FFHQ DiADM samples against ground-truth identity/attribute labels to measure actual training-set coverage. If IRS∞,a remains above 1 while label coverage is below 100%, or if the ratio shifts by more than 10% across extractors, the normalized score is an artifact of the measurement-gap correction rather than a bounded diversity measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.4 defines IRS∞,a = IRS∞,snth / IRS∞,real and states that the adjusted score can be above one. This is not a harmless edge case: Table 3 reports IRS∞,a = 1.51 for FFHQ and 1.08 for ChestX-ray14 for DiADM, meaning the metric claims these generators are 151% and 108% as diverse as the training set. That contradicts the paper's own premise in Sec. 3.1 that synthetic data should cover the training distribution and that IRS∞ represents 'the percentage of samples that the model can generate at its limit.' The cause is that IRS∞,real is not a measurement of full training diversity; it is the feature extractor's retrieval rate on a reference set, depressed by the measurement gap documented in Sec. 4.1. Dividing synthetic diversity by this depressed denominator can manufacture gains whenever synthetic features are less collapsed or differently distributed than real features. The headline 77% ceiling in Table 2 is computed with this same ratio, so the ceiling is not established as a statement about training-data diversity unless the normalization is shown to be valid across extractors and datasets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Image Retrieval Score (IRS), a diversity metric for generative models that counts how many training images are retrieved by synthetic queries in a pretrained feature space, and derives an estimator from the coupon collector problem with confidence intervals. Using IRS, the authors argue that current feature extractors are inadequate for diversity evaluation, report that state-of-the-art diffusion models on ImageNet do not exceed 77% of training-set diversity, and introduce Diversity-Aware Diffusion Models (DiADM), which conditions unconditional diffusion models on per-image features to improve diversity without sacrificing FID. An open-source Python package is provided.","tokens_in":21877,"tokens_out":8023,"duration_ms":77031,"significance":"If the metric is valid, IRS offers a more interpretable, statistically grounded diversity score than existing metrics, and the claim that current diffusion models cover only about three-quarters of the training distribution is an important empirical finding. The class-removal sanity check in Fig. 5 is a genuinely useful validation experiment, and the provided package and reproducible feature-extraction pipeline are strengths. However, the adjusted IRS can exceed 1, the confidence-interval formulas appear inconsistent, the metric depends on the chosen feature extractor, and the DiADM evaluation is confounded by the use of Inception features both for conditioning and for FID computation. These issues currently prevent the headline claims from being accepted as stated.","major_comments":[{"comment":"Equation (2) is mathematically inconsistent with the summation in Eq. (1). The sum of k terms each equal to 1 - ((Ntrain-1)/Ntrain)^n should be k times that value; as printed, Eq. (2) omits the factor k (presumably Ntrain) and gives a probability-like value rather than the expected number of learned images. If the intended quantity is the expected fraction of learned images, the equation should be normalized by Ntrain. This matters because Fig. 4 and Table 6 use the 'idealized scenario' as a reference; please correct the equation and ensure the plotted reference uses the corrected form.","section":"Eq. (2), Sec. 3.1"},{"comment":"The confidence-interval formulas appear to have the wrong optimization direction and the wrong summation limits. For a lower bound, one should find the smallest s such that the probability of observing at least Nlearned unique images is at least alpha_e; the arg max in Eq. (8) is the opposite of that. For the upper bound, the sum should run over k = 1..Nlearned (the probability of observing at most Nlearned unique images), not over k = 1..s. As written, the intervals reported in Sec. 4.2 (e.g., 86% [75%, 100%]) are not valid confidence intervals. Please re-derive these formulas and re-run the reported intervals.","section":"Eqs. (8)-(9), Sec. 3.3"},{"comment":"The adjusted IRS can exceed 1, and the paper's own interpretation that IRS represents 'the percentage of samples that the model can generate at its limit' is then broken. Table 3 reports IRS_inf,a = 1.51 for FFHQ and 1.08 for ChestX-ray14 with DiADM, and Sec. 3.4 excuses this only 'for low alpha values,' yet Table 3 uses the paper's standard sampling budget. Because the 77% ceiling in Table 2 is computed with the same adjusted ratio, the ceiling is not established as a bound on training-data diversity. Please report the unadjusted IRS_inf,snth and IRS_inf,real values separately and either demonstrate that the ratio is a valid, bounded diversity measure across alphas and extractors or reinterpret the adjusted score as a relative feature-space diversity index rather than a percentage of the training distribution.","section":"Sec. 3.4, Table 3"},{"comment":"The claim that IRS is 'hyperparameter-free' is misleading in light of the strong dependence on the feature extractor and distance metric. Table 6 shows IRS_real ranging from about 57% to 88% across extractors on the same datasets, and the supplementary table shows that the relative ranking of extractors changes between cosine and Euclidean distance. The paper selects SwAV based on consensus agreement (Tab. 1) but does not show whether the adjusted IRS values, the 77% ceiling, or the rankings in Table 2 are stable across different choices of F and P. Please add a sensitivity analysis of adjusted IRS over feature extractors and distance measures, or temper the 'hyperparameter-free' and 77% claims.","section":"Tab. 6, Sec. 4.1"},{"comment":"The DiADM evaluation is confounded by the fact that DiADM conditions on Inception-v3 features while FID is computed with the Inception-v3 feature space. The large FID improvements in Table 3 (e.g., 40.92 to 6.24 on FFHQ) may reflect the conditioning shortcut rather than genuine quality gains. Likewise, the diversity gain may be inflated because the model is explicitly trained to map training-set features to their source images, and IRS counts unique training-image retrievals under the same retrieval paradigm. Please evaluate DiADM with feature extractors not used for conditioning (for both FID and IRS), and include a baseline conditioned on random or low-information features to isolate the effect. Reporting memorization metrics would also address the concern acknowledged in Sec. 4.5.","section":"Sec. 3.5, Table 3"}],"minor_comments":[{"comment":"The paper contains several typos, e.g., 'Eucledian' (Sec. 3.2), 'quantize' for 'quantify' (Sec. 4.2), 'snyhtetic' (Sec. 3.4), 'accroding' (Sec. 4.2), and 'prove' for 'proof' (Fig. 11 caption).","section":"Throughout"},{"comment":"The symbol s is overloaded: in Eq. (5) it denotes Ntrain, while in Eqs. (7)-(9) it denotes the unknown true number of distinct images that the model can generate. Please use separate symbols to avoid confusion.","section":"Eq. (5) vs Eqs. (7)-(9)"},{"comment":"The phrase 'idealized scenario reaches IRS 7/3 = 96.98%' is unclear: 7/3 is the sampling ratio alpha, not an IRS value, and the numerical conversion to 96.98% does not follow from the stated Ntrain and Nsample values. Please clarify the computation.","section":"Sec. 12, Table 6 caption"},{"comment":"The legend distinguishes 'Theoretical limit' and 'Idealized'; the text says the idealized scenario follows Eq. (2), so the relationship between these two curves should be stated explicitly.","section":"Fig. 4"},{"comment":"The adjustment step in Sec. 3.4 is an assumption that dividing by IRS_real removes the measurement gap, but the paper does not provide a formal justification or a test of this assumption across feature extractors. Please add a short discussion or reference to the supplementary analysis beyond the class-removal experiment.","section":"Sec. 3.4"},{"comment":"Several references lack venue or publication details (e.g., [22] 'Classifier-Free Diffusion Guidance'), and the arXiv-only references should be marked as such consistently.","section":"Reference list"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the class-removal validation is a strong point. However, the adjusted metric's interpretability, the confidence-interval correctness, and the DiADM evaluation confound are load-bearing and need substantive fixes. The paper also overclaims 'hyperparameter-free' and the 77% ceiling; these claims should be softened unless robustness across feature extractors is demonstrated. I recommend major revision rather than rejection because the issues appear addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea—treating diversity as image retrieval and fitting a coupon-collector model—is genuinely useful, but the headline numbers (the 77% ceiling, DiADM scores above 1.0) rest on a normalization that is not justified, and the confidence-interval formulas look broken. I'd send it to a referee, but not with the current claims intact.\n\nWhat's new and good: IRS is more interpretable than FID or Recall, because it measures the fraction of training images retrieved at infinite sampling, with a statistical confidence interval. The class-removal experiment (Fig. 5) is the right kind of sanity check, and IRS tracks the true diversity well while Recall and Coverage saturate. The measurement-gap finding—common feature extractors retrieve far fewer real images than expected—is an important observation that also challenges FID-based evaluation. The authors are upfront that IRS depends on the feature extractor, and they release code.\n\nSoft spots, in order of severity. First, the adjusted score IRS∞,a = IRS∞,snth / IRS∞,real is supposed to eliminate the measurement gap, but it does not. Table 3 reports DiADM at 1.51 on FFHQ and 1.08 on ChestX-ray14, which the paper itself interprets as 'better than the real reference dataset.' That contradicts the Sec. 3.1 premise that synthetic data should cover the training distribution. The ratio can exceed one because the denominator is not the true diversity of the training set; it is the extractor's retrieval rate on a reference split, depressed by the measurement gap. Dividing by that number rescales but does not remove the gap unless the gap is a constant multiplicative factor, which is not shown. The headline 77% ceiling is computed with this same ratio, so it is not established as a statement about training-data diversity.\n\nSecond, Eqs. (8)-(9) are inconsistent as typeset. The sums and inequalities do not define valid lower or upper bounds; this needs a careful rewrite with a correct derivation.\n\nThird, DiADM conditions on per-image training features and is then evaluated by retrieval in the same feature space. That entangles the diversity gain with the conditioning mechanism, and the comparison is to the authors' own re-trained baseline, not to pretrained SOTA at matched quality. So 'without loss of image quality' is a claim about their compute-matched setup, not a general one.\n\nWho this is for: anyone evaluating generative models. The metric idea and the measurement-gap analysis deserve engagement even if the current numbers don't hold. Recommendation: send to peer review, but expect major revision and re-analysis of the adjusted metric, the confidence intervals, and the DiADM claims.","headline":"A useful retrieval-based diversity metric with a serious normalization problem; the headline numbers are not yet trustworthy, but the paper deserves a serious referee.","tokens_in":22424,"tokens_out":4934,"would_cite":false,"duration_ms":44005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims diversity in generative models is measurable through image retrieval, shows current diffusion models top out at 77% of training-data diversity, and presents DiADM to close the gap without quality loss.","keywords":["image diversity","diffusion models","Image Retrieval Score","generative model evaluation","coupon collector problem","mode collapse","feature extractors","pseudo-unconditional generation"],"falsifier":"Take a model trained on a dataset where 50% of training images are exact duplicates or where a held-out human can identify each synthetic image's source; if the adjusted IRS of such a deliberately half-collapsed model does not approach 50%, or if the same model's adjusted IRS changes rank when SwAV is replaced by DINOv2, the central claim that IRS measures diversity rather than extractor behavior is falsified.","tokens_in":21427,"feed_emoji":"🖼️","tokens_out":5873,"duration_ms":51574,"temperature":0.7,"pith_summary":"The paper tries to establish that the diversity of a generative model is a tractable, measurable quantity: how many distinct training images the model's outputs can stand in for. It packages this as the Image Retrieval Score (IRS), which counts the fraction of training images that are the nearest feature-space neighbor of at least one synthetic sample, with confidence intervals derived from the coupon collector problem. Using IRS, the paper argues that current feature extractors used in generative evaluation badly underestimate diversity (the 'measurement gap'), and that state-of-the-art diffusion models reach at most 77% of training-data diversity on ImageNet. It then introduces Diversity-Aware Diffusion Models (DiADM), which feed precomputed image features into an unconditional diffusion model as pseudo-labels, and reports that this increases IRS substantially on several datasets while also improving FID. A sympathetic reader would care because if true, diversity becomes as easy to measure and optimize as quality, rather than being hidden behind visually plausible samples.","feed_headline":"No diffusion model tops 77 percent of ImageNet training-data diversity","feed_subtitle":"A retrieval-based score exposes the gap; pseudo-label features close it without hurting image quality.","key_machinery":"The load-bearing object is the Image Retrieval Score (IRS), defined in Eq. (4) as $N_{\\mathrm{learned}}/N_{\\mathrm{train}}$, where a training image counts as learned if it is the nearest neighbor, in a pretrained feature space, of at least one synthetic image (Eq. (3)). The score is read through the coupon collector problem: drawing synthetic samples is modeled as drawing with replacement from the $N_{\\mathrm{train}}$ training images, so the probability of observing exactly $k$ distinct retrieved images is $P(k,n,s)=\\mathrm{Stir}(n,k)\\,s!/(s-k)!\\,s^{-n}$, evaluated in log space with Temme's asymptotic Stirling estimate to yield $\\mathrm{IRS}_\\infty$ and confidence bounds for infinite sampling. A normalization step divides synthetic IRS by real-data IRS to cancel the feature extractor's 'measurement gap'. The diversity-improving half of the paper is DiADM, which replaces unconditional placeholder labels with precomputed image features (pseudo-unconditional conditioning) so that each training instance behaves as its own class, disentangling fidelity from coverage.","core_discovery":"The central discovery claim is that diversity can be defined operationally as retrievability: a real training image is 'learned' when at least one synthetic image is closer to it, in a chosen feature space, than to any other training image; the model's diversity is the fraction of the training set that is learned, extrapolated to infinite sampling via a Stirling-number occupancy model. The paper asserts that every current feature extractor collapses the real data distribution, so a fair score must normalize synthetic diversity by the diversity measured between real and real reference data. On that basis it reports that no current state-of-the-art diffusion model surpasses 77% of training-data diversity, and that its DiADM module, which uses Inception features as pseudo-unconditional conditioning, raises this ceiling on ImageNet, FFHQ, ChestX-ray14, CelebV-HQ, and Dynamic while simultaneously improving FID.","pith_inferences":["If IRS is adopted, model rankings by diversity will become extractor- and reference-set-dependent; the paper's 77% ceiling should be read as a property of the chosen SwAV feature space and ImageNet reference, not a universal constant.","DiADM's design suggests a general template: any conditional model could be steered by features rather than labels, but when the features encode identity, the method may trade diversity for memorization unless a retrieval-based guardrail is applied.","The real-vs-real normalization assumes the measurement gap affects synthetic and real images identically; a testable implication is that a feature extractor that collapses synthetic features more than real ones would inflate adjusted IRS, so practitioners should validate the ratio on datasets with known diversity.","Coupling IRS to fairness framing suggests diversity metrics could be used as bias audits, a direction the paper only sketches."],"forward_implications":["With IRS, diversity can be monitored with far fewer samples than FID needs, and checkpoints can be rejected early when observed duplicates make low diversity statistically certain.","Because common feature extractors collapse real-data diversity, existing precision, recall, and coverage numbers understate the diversity problem; diversity claims should be re-checked on feature spaces that retrieve real images well.","If no model surpasses 77% diversity, then conditional guidance and scaling alone have not solved mode coverage, and diversity needs an explicit training signal.","DiADM's pseudo-unconditional features give unconditional diffusion models a way to be queried for diverse outputs without label supervision, and in the paper's tables the added diversity comes with lower FID, not higher.","Applied to text-to-image models with a balanced reference set, IRS turns fairness questions like gender bias into a measurable diversity deficit of roughly 50%."],"supporting_citations":[{"why":"Supplies the coupon collector model that grounds IRS's expected-diversity formula and confidence intervals.","marker":"[5]"},{"why":"Provides the asymptotic Stirling number estimate used to compute IRS in log space for large datasets.","marker":"[51]"},{"why":"Powers the feature-extractor search and the argument that evaluation encoders are flawed.","marker":"[49]"},{"why":"Provides the EDM-2 backbone and baselines that DiADM modifies and against which diversity gains are measured.","marker":"[24]"},{"why":"Implements precision and recall, the main baseline diversity metrics IRS is compared against.","marker":"[26]"},{"why":"Supplies density and coverage baselines used in the metric comparison.","marker":"[32]"},{"why":"Flagged the diversity issue in synthetic medical video and provides domain-specific re-identification features for IRS.","marker":"[14]"},{"why":"Provides the EchoNet synthetic ultrasound dataset on which IRS first quantifies the diversity gap.","marker":"[40]"}],"fun_headline_variants":["Diversity score exposes diffusion models' 77% real-data ceiling","New metric proves diffusion models miss 23% of real data diversity","Pseudo-label conditioning boosts image diversity without hurting quality","Retrieval-based score reveals diffusion models' diversity limits","DiADM: Taming diffusion diversity with pseudo-unconditional features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each synthetic image is assumed to have exactly one 'main component' training image, that component is the nearest neighbor in a pretrained feature space, and dividing synthetic by real IRS fully cancels the feature space's measurement gap; if any of these fails, IRS measures feature-space uniqueness rather than true distributional diversity.","fun_headline_variants_meta":{"raw":{"variants":["Diversity score exposes diffusion models' 77% real-data ceiling","New metric proves diffusion models miss 23% of real data diversity","Pseudo-label conditioning boosts image diversity without hurting quality","Retrieval-based score reveals diffusion models' diversity limits","DiADM: Taming diffusion diversity with pseudo-unconditional features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3687,"prompt_tokens":995,"completion_tokens":2692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2606}},"tokens_in":611,"tokens_out":2692,"duration_ms":17886,"temperature":1.0,"reasoning_tokens":2606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:27:37.728870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model trained on a dataset where 50% of training images are exact duplicates or where a held-out human can identify each synthetic image's source; if the adjusted IRS of such a deliberately half-collapsed model does not approach 50%, or if the same model's adjusted IRS changes rank when SwAV is replaced by DINOv2, the central claim that IRS measures diversity rather than extractor behavior is falsified.","supporting_citations":[{"cited_title":"Baum and Patrick Billingsley","cited_arxiv_id":null,"evidence_quote":"Supplies the coupon collector model that grounds IRS's expected-diversity formula and confidence intervals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the asymptotic Stirling number estimate used to compute IRS in log space for large datasets."},{"cited_title":"Improved Precision and Recall Metric for Assessing Generative Models","cited_arxiv_id":null,"evidence_quote":"Implements precision and recall, the main baseline diversity metrics IRS is compared against."},{"cited_title":"Reliable Fidelity and Diversity Metrics for Generative Models","cited_arxiv_id":null,"evidence_quote":"Supplies density and coverage baselines used in the metric comparison."},{"cited_title":"EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing","cited_arxiv_id":"2406.00808","evidence_quote":"Provides the EchoNet synthetic ultrasound dataset on which IRS first quantifies the diversity gap."}],"review_version":1}