{"id":"ff40d1ca-3c79-4f2b-b276-490a5d3e7345","arxiv_id":"2502.03668","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A comprehensive survey that organizes privacy attacks and privacy/utility metrics for GANs and VAEs into new taxonomies, but with a non-transparent paper-selection process and several metric errors.","lead":"This paper surveys 100 research papers on privacy-preserving generative models such as GANs and VAEs, and proposes taxonomies of privacy and utility metrics. It is a reference guide for researchers choosing metrics, though it contains notable equation errors.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Several core metric equations are wrong (e.g., Eq. 26 PSNR is inverted), so the survey's practical claim of helping researchers select evaluation measures is compromised.","rationale":"The reader's weakest assumption was the undocumented paper-selection process. I agree that this is a genuine limitation, but it is secondary: even with perfect selection, the survey's central claim to help researchers 'identify suitable metrics' fails if the metric equations are wrong. The errors are concrete, checkable, and pervasive across the taxonomy (PSNR, silhouette, MAD, SS_total). They affect the core reference value of the paper, not just its historical claims. I give credit for the breadth of coverage and the taxonomy structure, which are valuable. The concern does not warrant rejection because the errors are localized and fixable; the taxonomy can stand once equations are corrected. The reader already reached CONDITIONAL; my concern reinforces that verdict rather than changing it, so verdict_should_be is UNCHANGED. I partially agree with the reader's weakest_assumption: the selection process is indeed undocumented, but the equation errors are more directly load-bearing for the central practical claim.","tokens_in":30401,"tokens_out":4631,"duration_ms":39642,"concrete_test":"Recompute PSNR for a simple 2×2 grayscale image pair with MSE=100 and MAX=255 using the paper's Eq. (26): the result is 10·log10(100/255²) ≈ −28.1 dB, whereas the standard formula gives +28.1 dB. If the paper's equation yields a negative value for a valid image pair, the formula is inverted, confirming that the survey's metric definitions are unreliable and the reference-map claim is weakened. Independently, re-derive Eq. (7) for a point closer to its own cluster than to the nearest neighbor; the paper's sign is opposite to the standard silhouette score.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central value is a reliable map of privacy and utility metrics that researchers can use to select evaluation measures. That value depends on the metric definitions being correct. Section 4.2.2, Eq. (26) defines PSNR as 10·log10(MSE/MAX²); the standard definition is 10·log10(MAX²/MSE). For any typical image pair (MSE < MAX²), the paper's formula returns a negative dB value, reversing the metric's semantics and making higher PSNR indicate worse quality. Section 4.1.3, Eq. (7) defines the silhouette score as (a−b)/max(b,a), but standard silhouette is (b−a)/max(a,b) where a is intra-cluster distance and b is nearest-cluster distance; the paper swaps a and b, flipping the sign and thus the interpretation of cluster quality. Section 4.1.2, Eq. (3) defines MAD as median(|y_i − median(y)|), which is the median absolute deviation from the overall median, not the median absolute difference between actual and predicted values as the text claims. Section 4.1.2, Eq. (6) labels the average squared error as SS_total, which is actually SSE. These are not superficial typos: they invert or misstate the very quantities the survey is meant to clarify. A researcher selecting metrics from this survey would draw incorrect conclusions, e.g., treating a low PSNR as high fidelity. The taxonomy structure and paper coverage may be sound, but the reference value of the 'novel taxonomies' is undercut by incorrect formal definitions. This is the most load-bearing concern because it directly threatens the paper's practical utility, the claimed contribution of guiding metric selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper surveys privacy and utility evaluation for generative models (GANs and VAEs). It proposes taxonomies for privacy attacks, privacy metrics, and utility metrics; reviews 100 publications; and discusses open challenges and future research directions. The central claims are that this is the first survey to systematically categorize privacy and utility perspectives of GANs and VAEs and that the proposed taxonomies provide a structured map for researchers selecting evaluation measures.","tokens_in":30684,"tokens_out":6898,"duration_ms":60669,"significance":"If corrected, this survey would be a useful reference: the taxonomies in Figures 2 and 6 and the data-type mapping in Table 2 organize a large body of work, and the metric-by-metric attributions are checkable against the cited literature. The paper does not ship code or machine-checked proofs, but it does provide a broad, manually curated coverage of 100 papers. The main value depends on the formal definitions being correct, because the stated purpose is to help researchers select and interpret privacy and utility metrics; several of the printed formulas are inconsistent with standard definitions and must be corrected before the reference value of the survey is reliable.","major_comments":[{"comment":"Eq. (26) defines PSNR = 10 log10(MSE/MAX^2), which is inverted with respect to the standard definition PSNR = 10 log10(MAX^2/MSE). For typical images with MSE < MAX^2, the printed formula returns negative values, reversing the semantics of the metric; since Section 5.2 and Table 2 recommend PSNR for image fidelity, a reader following the survey would misinterpret PSNR values. This equation should be corrected and the surrounding discussion updated.","section":"§4.2.2, Eq. (26)"},{"comment":"The silhouette score formula and its variable definitions are inconsistent with the standard definition. The standard silhouette score is (b-a)/max(a,b), with a the mean intra-cluster distance and b the mean nearest-cluster distance; Eq. (7) prints (a-b)/max(b,a) and then defines b as the within-cluster distance and a as a between-cluster distance. As printed, a reader using the standard notation would obtain the wrong sign, and the text's 'average distance between all clusters' is not the nearest-cluster distance used by the silhouette score. Please align the notation with the standard definition or state the swapped convention explicitly.","section":"§4.1.3, Eq. (7)"},{"comment":"The MAD metric in Eq. (3) is the median absolute deviation from the median of the target values, median(|y_i - median(y)|). This quantity does not compare actual target values with predicted values, despite the text's claim that MAD is a regression utility metric comparing synthetic predictions with real targets. If the intended quantity is the median absolute difference between y_i and y_i', the formula should involve both y_i and y_i'; as written it cannot serve the stated purpose.","section":"§4.1.2, Eq. (3)"},{"comment":"Eq. (6) defines SStotal as (1/n) sum (y_i - y_i')^2, which is the mean squared error, not the total sum of squares of the observed data. In the standard R^2 formula, SStotal = sum (y_i - bar{y})^2, and the denominator in Eq. (5) must be the total variance of the target variable. As printed, Eqs. (5) and (6) do not define R^2 and would mislead a reader computing goodness of fit for synthetic data.","section":"§4.1.2, Eqs. (5)-(6)"},{"comment":"The paper's comprehensiveness claim is not supported by a documented selection process: Section 5 states that 'Starting with approximately 1200 papers, we identified 100 research publications for in-depth analysis,' but the survey provides no search strategy, database sources, inclusion/exclusion criteria, screening steps, or inter-rater procedure. Without this information, the 'first comprehensive survey' claim and the completeness of the taxonomies in Figures 2 and 6 cannot be assessed. Please add a methodology subsection or temper the claims accordingly.","section":"§5"}],"minor_comments":[{"comment":"The text says 'The range of CCS is −1 to 1' but the acronym is CSS; the term 'clustering silhouette score' should be defined before Eq. (7) is used.","section":"§4.1.3"},{"comment":"Eq. (22) uses the undefined symbol 'xbar' in the displayed formula; replace it with bar{x} or define it in the notation list.","section":"§4.2.1, Eqs. (21)-(22)"},{"comment":"'Frobennius norm' should be 'Frobenius norm'.","section":"§4.2.2, Eq. (27)"},{"comment":"Eq. (4) sums from i=0 to n-1 while the other regression equations sum from i=1 to n; the indexing should be made consistent.","section":"§4.1.2, Eq. (4)"},{"comment":"The description of the true acceptance rate (TAR) as 'the ratio of true positive identifications to false positive identifications' is not the standard definition; TAR is normally the true positive rate. Please correct or qualify this description.","section":"§3.1.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a survey venue, but its reference value depends on the correctness of the metric definitions, and several load-bearing equations need correction. I do not see a circularity problem in the self-citation [101]; it is used only to support a general recommendation on average and worst-case metrics. The undocumented paper-selection process is a substantive survey-quality concern, but it can be addressed by adding a methodology subsection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a useful map, not a discovery. It is the first survey I know that puts privacy and utility metrics for GANs and VAEs side by side, and that organization alone helps people entering the area. The taxonomy figures are clear, the attack classification is sensible, and the discussion of open challenges is honest. It covers 100 papers and cites prior surveys fairly. If you need a starting point for metric selection, this is a reasonable one.\n\nThe soft spots are real. Section 4.2.2 Eq. (26) defines PSNR as 10·log10(MSE/MAX²). Standard PSNR is 10·log10(MAX²/MSE). The printed formula returns negative dB for ordinary images, reversing the metric's meaning. Section 4.1.3 Eq. (7) has the silhouette score sign swapped: standard is (b−a)/max(a,b) with a intra-cluster and b nearest-cluster distance. As written, high scores mean poor clustering. Section 4.1.2 Eq. (3) calls median(|y_i − median(y)|) the median absolute difference between actual and predicted values, but that formula is median absolute deviation from the median. Eq. (6) labels the average squared error as SS_total, which is actually SSE. These are not cosmetic typos in a survey whose stated value is helping researchers select evaluation measures. A reader copying Eq. (26) would draw incorrect conclusions.\n\nThe other weakness is the undocumented paper-selection process. Section 5 says they started with ~1200 papers and identified 100 for in-depth analysis, but no search strategy, inclusion/exclusion criteria, or screening procedure is given. The 'first work' and 'comprehensive' claims cannot be checked. This is less damaging than the formula errors but should be fixed.\n\nNone of this is fatal to the survey's organizational contribution. The taxonomies and the prose around them are still useful. But the survey should not be cited as a reference for metric definitions until the equations are corrected and the methodology is documented.\n\nI would send it to peer review with those conditions. A good referee can help the authors fix the equations and add a methods paragraph. For a newcomer to privacy-preserving generative models, the corrected version would be worth reading. As it stands, use it for the map, not for the formulas.","headline":"A useful but uneven survey map: the taxonomy and coverage are genuinely helpful, while several metric formulas are wrong and the paper-selection method is undocumented.","tokens_in":31267,"tokens_out":2550,"would_cite":false,"duration_ms":24692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey maps the privacy and utility metrics used for GANs and VAEs into two novel taxonomies, based on 100 publications.","keywords":["generative adversarial networks","variational autoencoders","differential privacy","membership inference attacks","privacy metrics","utility metrics","synthetic data","privacy-preserving machine learning"],"falsifier":"Reproduce the selection process from the paper's stated scope and sources; if it does not converge on the same 100 papers, or if a systematically chosen set of additional papers on privacy-preserving diffusion models yields metrics that fit none of the taxonomy's categories, the survey's completeness claim fails.","tokens_in":30175,"feed_emoji":"🔒","tokens_out":6442,"duration_ms":56789,"temperature":0.7,"pith_summary":"This survey sets out to be the first systematic map of both privacy and utility metrics for generative models, specifically GANs and VAEs, built from an in-depth analysis of 100 research publications. It groups privacy attacks into four families and then arranges privacy and utility metrics into taxonomies, with the practical goal of letting a researcher choose evaluation measures knowingly. The paper argues that no earlier survey drew the same two-sided picture, and that this structure exposes gaps such as the absence of formal guarantees in attack-based metrics and the lack of a one-size-fits-all utility metric. If the map holds, it gives newcomers a shared vocabulary for comparing privacy-preserving generative models.","feed_headline":"100 papers, one map: privacy and utility metrics for GANs and VAEs","feed_subtitle":"A survey of 100 studies sorts the ways synthetic data leaks information and the ways we measure the damage.","key_machinery":"The load-bearing device is the classification system itself: the attack taxonomy in Figure 1 divides privacy attacks by what they target, the privacy metric taxonomy in Figure 2 groups metrics into attack-based, generalization-based, and differential-privacy-based families, and the utility metric taxonomy in Figure 6 groups metrics into specific-task and fidelity families. Each metric gets a location in this tree and a set of neighbouring alternatives, which is what allows the survey to compare trade-offs and to spot missing categories.","core_discovery":"On the paper's own terms, the discovery is organizational rather than experimental: privacy and utility evaluation in generative models can be systematically categorized, and doing so reveals what existing measurements do and do not cover. The privacy side distinguishes four attack levels, namely training data, attribute, model, and identification-based, and three metric families: attack-based, generalization-based, and differential-privacy-based. The utility side distinguishes task-specific utility, covering classification, regression, and clustering, from fidelity, which is further split into distributional and individual-sample comparisons. The authors present these taxonomies as novel, based on 100 papers, and use them to compare the pros and cons of individual metrics and to identify open research challenges.","pith_inferences":["The same two-sided taxonomy could plausibly be applied to synthetic data from diffusion models, but the model-level attack branch would likely need extension because diffusion models expose different components than GANs and VAEs.","The paper's comparison suggests a practical selection rule, namely use a differential-privacy metric when a formal guarantee is required, an attack-based metric when the adversary model is concrete, and a fidelity metric when downstream use is the goal; the authors do not explicitly state this rule.","A reproducible version of the selection process would let the taxonomy be updated as new attacks and metrics appear, turning it from a static survey into a living map."],"forward_implications":["A newcomer can locate an attack or metric in the tree and immediately see which family it belongs to and what assumptions it carries.","Researchers can compare GAN- and VAE-based privacy results more fairly, since the taxonomy groups metrics by what they measure rather than by paper-specific names.","The identified trade-off between formal differential-privacy guarantees and practical attack-based measurement becomes a standard axis for evaluating synthetic data.","The utility taxonomy gives downstream users a checklist: classification, regression, and clustering metrics for task utility, plus distributional and individual distance metrics for fidelity.","The open-challenge discussion points to concrete next steps, such as robust distance-based privacy metrics and fairness metrics, for researchers entering the area."],"supporting_citations":[{"why":"Introduces the GAN framework that the survey's attack and utility discussion builds on.","marker":"[40]"},{"why":"Introduces VAEs, the other model family covered by the survey.","marker":"[59]"},{"why":"Defines differential privacy, the central guarantee underlying the DP-based metric family.","marker":"[29]"},{"why":"Demonstrates attribute inference and re-identification risks in synthetic data, anchoring the attack taxonomy.","marker":"[94]"},{"why":"Provides a taxonomy of membership inference attacks against generative models, a direct precursor to the survey's attack classification.","marker":"[17]"},{"why":"Supplies DPSGD, the dominant mechanism behind the differentially private GAN and VAE training approaches surveyed.","marker":"[1]"},{"why":"Introduces PATE, the teacher-student mechanism that supports the PATE-GAN family.","marker":"[86]"},{"why":"Gives the PATE-GAN framework used to illustrate noise addition through teacher discriminators.","marker":"[55]"},{"why":"Introduces the FID metric, a central fidelity measure in the utility taxonomy for image data.","marker":"[45]"}],"fun_headline_variants":["Categorizing privacy and utility metrics for GANs and VAEs","100 papers, one taxonomy: GAN privacy and utility","New taxonomies for GAN privacy and utility evaluation","Privacy and utility, neatly categorized for GANs and VAEs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The map's completeness rests on the undocumented selection of 100 papers from roughly 1200; if that selection is biased or incomplete, the taxonomies may miss important metrics or attacks even though individual descriptions may still be accurate.","fun_headline_variants_meta":{"raw":{"variants":["Categorizing privacy and utility metrics for GANs and VAEs","100 papers, one taxonomy: GAN privacy and utility","New taxonomies for GAN privacy and utility evaluation","Privacy and utility, neatly categorized for GANs and VAEs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001954,"raw_usage":{"total_tokens":7550,"prompt_tokens":765,"completion_tokens":6785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":381,"completion_tokens_details":{"reasoning_tokens":6713}},"tokens_in":381,"tokens_out":6785,"duration_ms":45572,"temperature":1.0,"reasoning_tokens":6713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:08:43.654755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the selection process from the paper's stated scope and sources; if it does not converge on the same 100 papers, or if a systematically chosen set of additional papers on privacy-preserving diffusion models yields metrics that fit none of the taxonomy's categories, the survey's completeness claim fails.","supporting_citations":[{"cited_title":"Stochastic gradient vb and the variational auto-encoder","cited_arxiv_id":null,"evidence_quote":"Introduces VAEs, the other model family covered by the survey."},{"cited_title":"Synthetic Data – Anonymisation Groundhog Day","cited_arxiv_id":null,"evidence_quote":"Demonstrates attribute inference and re-identification risks in synthetic data, anchoring the attack taxonomy."},{"cited_title":"Scalable Private Learning with PATE","cited_arxiv_id":null,"evidence_quote":"Introduces PATE, the teacher-student mechanism that supports the PATE-GAN family."},{"cited_title":"PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees","cited_arxiv_id":null,"evidence_quote":"Gives the PATE-GAN framework used to illustrate noise addition through teacher discriminators."}],"review_version":1}