{"id":"9f5d9da0-2503-4bf5-bb56-70f36afa29c7","arxiv_id":"2507.11324","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A review that lists 17 privacy metrics for synthetic data with their formulas and assumptions, rescaling each to a common privacy-risk direction.","lead":"This paper catalogues 17 privacy metrics used to evaluate synthetic data generators, presenting each metric's assumptions and mathematical formula, then rescaling all scores to a single 0-to-1 privacy-risk scale. A smart generalist reading it will learn that one privacy number can hide very different attacker models, but also that several formulas in the paper are unreliable as written.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Identifiability (ID) score is not correctly formulated: Eq. (19) divides by the entropy weight w(a,v), so high-entropy attributes are amplified, exactly opposite to the stated weighting rationale in §2.2.5.","rationale":"I agree with the reader's REJECT verdict, but only partially on the load-bearing concern: the reader's weakest_assumption is the rescaling of DVP, Auth, DCR, etc., whereas I believe the decisive problem is the internal inconsistency in the ID weighting equations. The rescaling transforms are disclosed and could be debated; Eq. (19) is mathematically falsifiable against the authors' own stated intent. The central claim—correct formulations for all 17 metrics—requires every metric's equations to be well-defined and direction-consistent. ID fails that. Because the error inverts the weighting, it is not a cosmetic typo: it changes which attributes dominate the distance and hence the ID score. This is sufficient to keep the verdict at REJECT. I do not see a scenario where this concern is resolved without a correction to the text or code.","tokens_in":10182,"tokens_out":7227,"duration_ms":83376,"concrete_test":"Implement both variants on a small dataset with two attributes, one low-entropy (nearly constant) and one high-entropy (uniform over many values), with ε=1e-8. Variant A uses Eq. (19) as printed, ŷ[a]=y[a]/w(a,y[a])+ε; Variant B uses the weighting stated in the text, ŷ[a]=y[a]·w(a,y[a])+ε. For each real record, compare d_E(ŷ, nearest synthetic ŷ) under the two variants and compute the resulting ID scores from Eq. (20). If Variant A gives a higher ID score when high-entropy attributes dominate, Eq. (19) is doing the opposite of the paper's stated rationale. Also inspect the PrivEval source code (linked in §Artifact Availability): if it implements ŷ[a]=y[a]·w(a,y[a]), then Eq. (19) does not match the released tool and must be corrected; if it matches Eq. (19), then the textual rationale in §2.2.5 is wrong.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central promise is that the 17 metric formulations are correct, so practitioners can use the catalog as a reference. The ID-score section (§2.2.5) breaks this promise internally. The text says low entropy makes an attribute useful for re-identification, so low-entropy attributes should get a high weight, and it defines w(a,v)=1/(H(a,v)+ε) exactly that way. But Eq. (19) then constructs the weighted values as ŷ[a]=y[a]/(w(a,y[a])+ε). With w≈1/(H+ε), this division is multiplication by H+ε: a high-entropy (low-weight) attribute is scaled up, and a low-entropy (high-weight, discriminating) attribute is scaled down. The subsequent statement that 'a low weight means a low contribution to the distance measure' is therefore false for the formula as printed. Since every ID distance in Eq. (20) is computed on these weighted vectors, the metric measures the opposite weighting scheme from the one the authors describe, and the catalog entry for ID is not a correct formulation. Related internal inconsistencies elsewhere (e.g., Eq. (30) defines h(a) as max−max=0; Eq. (24)'s argmin uses the wrong variable) make clear this is not an isolated proofreading issue. The abstract's claim is unsupported until these definitions are reconciled with the linked tool's actual implementation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to be a reference catalog of privacy metrics for privacy-preserving synthetic data generation (PP-SDG). It defines a common setup (real data Y, synthetic data Z), imposes a unified [0,1] privacy-risk scale in which 0 means complete privacy and 1 means no privacy, and then presents assumptions and mathematical formulations for 17 metrics, divided into simulation-based and distance-based families. Many of the metrics are rescaled or transformed from their original definitions to fit the unified scale, and the authors link the catalog to their PrivEval implementation.","tokens_in":10469,"tokens_out":7608,"duration_ms":97753,"significance":"If the formulations were correct, the paper would fill a useful gap: practitioners could compare privacy metrics on a common scale and would have a single reference for the assumptions behind each metric. The paper also has practical value in making its code available and in being explicit about rescaling choices, which is more than most metric catalogs do. However, the central claim is that the presented formulas are correct, and the manuscript contains several load-bearing formula errors that contradict the accompanying text. Because the paper's product is the mathematical formulation itself, these errors directly undermine its main contribution. The paper should not be accepted until every equation is checked against the original sources and the linked implementation; with correction, the catalog could be a useful reference.","major_comments":[{"comment":"The Identifiability (ID) score weighting is inverted. Eq. (18) defines w(a,v)=1/(H(a,v)+ε), so low-entropy attributes receive high weight. Eq. (19) then sets ŷ[a]=y[a]/(w(a,y[a])+ε), which is division by the weight and therefore multiplies by H(a,y[a])+ε. This amplifies high-entropy attributes and damps low-entropy attributes, exactly the opposite of the stated rationale in §2.2.5. The sentence \"a low weight means a low contribution to the distance measure\" is false for the formula as printed, and because Eq. (20) computes all ID distances on these weighted vectors, the metric as formulated measures the opposite weighting scheme from the one the authors describe.","section":"§2.2.5, Eq. (19)"},{"comment":"The Hitting Rate threshold h(a) is identically zero as printed. The equation reads h(a)=max_{y∈Y} y[a] − max_{y∈Y} y[a] / 30, so the numerator is a maximum minus the same maximum, giving h(a)=0 for every attribute. This makes the \"similar values\" condition in C_HitR an exact-equality condition on continuous attributes, which contradicts the text's description of attribute-dependent thresholds. The formula should presumably be (max_y y[a] − min_y y[a])/30 or a similar range-based expression, but as written it is not a usable threshold.","section":"§2.2.13, Eq. (30)"},{"comment":"The F1 formula for Attribute Inference Risk is malformed. As typeset, the expression does not reduce to the standard F1 = 2TP/(2TP+FP+FN), and the nesting of fractions is not well defined. Since AIR is defined as a weighted average of this F1 value, the quantitative behavior of the entire AIR metric is unclear from the printed formula. This needs to be corrected and verified against the source definition in [8].","section":"§2.1.3, Eq. (6)"},{"comment":"The set C_DCR has an ill-typed argmin. The equation writes z = arg min_{y∈Z} d_E(Φ_k(y), Φ_k(z)), but z is the variable being defined and y is the outer quantifier; the minimization should be over z∈Z for each fixed y. As written, the set cannot be constructed as intended. This is not a mere typo, because DCR is then computed as an average over this set, so the formula does not define a computable metric.","section":"§2.2.8, Eq. (24)"},{"comment":"The NNAA score is defined as a sum of two probabilities rather than their average. The equation sums the fraction of real points closer to a synthetic point than to a real point with the fraction of synthetic points closer to a synthetic point than to a real point. Since each fraction is in [0,1], the sum can exceed 1, and subtracting it from 1 in Eq. (28) can produce negative values. This violates the paper's own Definition 2 and contradicts the text's statement that 0.5 corresponds to an adversary with equal chance of guessing correctly. The intended definition should be the average of the two probabilities.","section":"§2.2.10, Eq. (28)"},{"comment":"The rescaling of MDCR as σ(MDCR) cannot produce values close to 0 for any nonnegative distance ratio. MDCR is a ratio of distances and is therefore nonnegative, and σ(x)≥0.5 for x≥0. The text states that an MDCR' score close to 0 means low re-identification risk, but the smallest possible value is σ(0)=0.5. If the goal is a rescaling that spans the full [0,1] range, a transformation such as σ(log(MDCR)) would be needed, not the one printed in Eq. (27).","section":"§2.2.9, Eq. (27)"}],"minor_comments":[{"comment":"In the NNDR formula, the distances d_E(y',z) and d_E(y'',z) should be d_E(Φ_k(y'),Φ_k(z)) and d_E(Φ_k(y''),Φ_k(z)) to be consistent with the mapping defined in Eq. (22). Also, the text refers to \"NDRR\" where \"NNDR\" is meant.","section":"§2.2.7, Eq. (23)"},{"comment":"The DCR' transformation uses log(DCR), which is undefined when DCR=0. The paper should either restrict the domain or add a small epsilon inside the logarithm, and should state how zero distances are handled.","section":"§2.2.8, Eq. (25)"},{"comment":"Definition 2 says that if Z=Y, a well-designed privacy metric should yield 1, but NSND in Eq. (21) gives 0 in that case because all nearest-neighbor distances are zero. The paper should acknowledge that distance-based metrics are not aligned with this expectation, or adjust the framing of Definition 2.","section":"Definition 2, §2.2.6"},{"comment":"There are several typographical and language issues: \"LigtGBM\" should be \"LightGBM\", \"neigbour\" should be \"neighbor\", \"distinct which real individual\" should be \"distinguish which real individual\", and \"a in a smaller vector space\" should be \"in a smaller vector space\".","section":"Throughout"},{"comment":"The text defining AIR says \"quantifies uses the F1 score,\" which is a grammatical error; it should read \"quantifies the attack success using the F1 score.\"","section":"§2.1.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a reference document for the authors' PrivEval tool, and the number of equation-level errors is high for a submission whose main contribution is the correctness of those equations. I would encourage the editor to ask the authors to validate every formula against the original cited papers and the actual PrivEval implementation, and to add a supplementary table or script that confirms each formula reproduces the reported behavior. The errors are individually fixable, but their prevalence makes a simple copyedit insufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main takeaway: the paper is a catalog of 17 privacy metrics with the goal of making their mathematical formulations precise enough for practitioners to use. That is a genuinely useful goal, and the structure is sensible: each metric gets a threat model, an equation, and a rescaling to a common 0-1 privacy-risk scale. The survey coverage is good, and the authors clearly know the synthetic-data literature. The companion PrivEval tool is a nice touch and gives readers somewhere to check the implementation.\n\nUnfortunately, the formulas are not reliable, and since the abstract explicitly promises 'correct assumptions and mathematical formulations,' that is a load-bearing problem. The stress-test note is right: in Section 2.2.5, the weight w(a,v) = 1/(H+ε) is defined so low-entropy attributes get high weight, but Eq. (19) divides by that weight, so high-entropy attributes end up amplified and low-entropy ones shrunk—exactly the opposite of the stated rationale. That is not a typo in a rare metric; it affects every ID distance in Eq. (20).\n\nThe other errors the reader flagged are real too. Eq. (30) defines h(a) as max minus max, which is identically zero. Eq. (6) is a malformed F1 expression that does not reduce to the standard formula. Eq. (24) has a variable mistake in the argmin (min over y in Z). These are exactly the places where a reference catalog has to be perfect. Also worth noting but less severe: the rescaling choices like DVP' = 1-DVP and Auth' = 1-Auth are monotone flips, which are fine for direction but do not by themselves guarantee comparability across datasets; the paper could say more about that. The fixed thresholds (like CVP's 0.2) appear without justification, but that is inherited from the source papers.\n\nOn the positive side, the threat-model taxonomy and the figures are clear, Definition 2 is reasonable, and the authors are honest about which rescalings they introduced. Nothing here looks like bad faith; it reads like a draft that was not checked equation-by-equation.\n\nWho is this for? Practitioners who want to compare metric behavior on their own data and researchers building evaluation frameworks. They should use the original papers and the tool, not these equations, until the errors are fixed.\n\nMy call: the catalog deserves a serious referee and a major revision, not a desk rejection. With a careful pass over every equation and a check against the tool's implementation, it could become a genuinely useful reference. As submitted, I would not cite it.","headline":"A useful catalog idea undermined by a cluster of formula errors, including an inverted weighting scheme in the ID score; fixable, but not trustworthy as submitted.","tokens_in":11019,"tokens_out":1987,"would_cite":false,"duration_ms":21960,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the 17 privacy metrics used to evaluate synthetic data can be restated under a single definition, with explicit attack assumptions and rescaling to a common [0,1] privacy-risk scale.","keywords":["privacy-preserving synthetic data","privacy metrics","differential privacy","re-identification risk","membership inference","attribute disclosure","synthetic data evaluation","privacy risk scale"],"falsifier":"Compute both the raw and rescaled versions of DVP on a suite of synthetic datasets with known re-identification rates; if $1-\\mathrm{DVP}$ does not track the fraction of real records an adversary can actually re-identify, the rescaling is an artifact rather than a transparency fix.","tokens_in":9987,"feed_emoji":"🔒","tokens_out":6381,"duration_ms":74804,"temperature":0.7,"pith_summary":"Privacy-preserving synthetic data generation aims to release synthetic datasets that hide individuals, but the standard guarantee, differential privacy, gives a budget $\\varepsilon$ that is hard for non-experts to interpret. This paper argues that the many privacy metrics proposed to make this risk concrete can be presented as a single coherent catalog: 17 metrics, each defined by explicit assumptions about the adversary and by a mathematical formula. To make them comparable, the paper fixes a common scale on which 0 means complete privacy and 1 means no privacy, and it rescales several published metrics that originally pointed in the opposite direction or had unbounded range. If the catalog is right, practitioners can choose a metric by matching the attack it models and read its output directly as a privacy-risk score.","feed_headline":"17 privacy metrics, one 0-to-1 privacy-risk scale","feed_subtitle":"A formal catalog restates synthetic-data privacy metrics under one definition, so scores can be compared directly.","key_machinery":"The load-bearing object is Definition 2, the unified privacy metric $p: \\mathcal{D}\\times\\mathcal{D}\\to[0,1]$ with 0 = complete privacy and 1 = no privacy, together with the two-component computation pattern: simulation-based metrics build a guess relation $G\\subset Y\\times Z$ and an oracle-corrected relation $C\\subseteq R$, then score the attack; distance-based metrics skip the adversary and compute a score directly from nearest-neighbour distances. The paper's rescaling identities — e.g., $1-\\mathrm{DVP}$, $1-\\mathrm{Auth}$, $1-\\mathrm{NNAA}$, $1-\\sigma(\\log \\mathrm{DCR})$, and $\\sigma(\\mathrm{MDCR})$ — are the mechanisms that force the published metrics onto the common scale.","core_discovery":"The paper's central claim is that 17 published privacy metrics used in privacy-preserving synthetic data generation can be stated on a common formal footing: each is a function $p(Y,Z)$ from a real dataset and a synthetic dataset to $[0,1]$, with 0 meaning complete privacy and 1 meaning no privacy. To achieve this, the paper restates the metrics in two families — simulation-based, where an adversary's guessed matches $G$ are checked against correct matches $C$ by an oracle, and distance-based, where distances between real and synthetic records are analysed directly — and it supplies rescaled versions of metrics whose original direction or range did not fit, such as $\\mathrm{DVP}' = 1-\\mathrm{DVP}$, $\\mathrm{Auth}'=1-\\mathrm{Auth}$, and $\\mathrm{DCR}' = 1-\\sigma(\\log \\mathrm{DCR})$. The result is a reference catalog that pairs each metric with the attack assumptions it embodies.","pith_inferences":["A natural next step the paper does not take is to check whether the 17 rescaled scores actually move together on benchmark datasets when the privacy risk changes; if they do not, the common scale is formal rather than behavioral.","Because the rescaling maps include nonlinear ones like $1-\\sigma(\\log \\mathrm{DCR})$, two datasets can be ordered one way by the raw DCR and the opposite way by $\\mathrm{DCR}'$; whether this flips practical privacy decisions is an open empirical question.","The taxonomy suggests a selection rule the paper leaves implicit: simulation-based metrics suit threat models with a concrete adversary, while distance-based metrics suit quick audits without a specified adversary."],"forward_implications":["A practitioner can now look up any of the 17 metrics, see the attack it assumes, and read its score as a privacy risk between 0 and 1 without re-deriving the formula.","The explicit assumptions make mismatches visible: for example, ZCAP and GCAP only apply to categorical sensitive attributes, while AIR is the variant that adds continuous attributes with a 10% tolerance.","Rescaled metrics such as $\\mathrm{DVP}'=1-\\mathrm{DVP}$ and $\\mathrm{Auth}'=1-\\mathrm{Auth}$ let metrics that originally measured distance or authenticity be compared in the same direction as the rest of the catalog.","The division into simulation-based and distance-based metrics gives a shared vocabulary for discussing which privacy attack a score is actually quantifying."],"supporting_citations":[{"why":"Defines ZCAP and GCAP, the categorical attribute-disclosure baselines whose match-and-correct-match structure the review formalizes.","marker":"[3]"},{"why":"Supplies the Attribute Inference Risk (AIR) metric, including its F1 scoring and entropy-based tuple weighting.","marker":"[8]"},{"why":"Supplies the family of distance and classifier-based metrics (CRP, CVP, DVP, D-MLP, Auth, ID, NSND) that form the largest group in the catalog.","marker":"[5]"},{"why":"Introduces NNDR, DCR, and HiddenRate, the metrics that use lower-dimensional embeddings and the rescaling with the sigmoid of log DCR.","marker":"[2]"},{"why":"Supplies the membership-inference and nearest-neighbour metrics MDCR, NNAA, MIR, and HitRate.","marker":"[4]"},{"why":"The companion evaluation tool in which the 17 metrics are implemented.","marker":"[6]"}],"fun_headline_variants":["17 privacy metrics unified on a 0-to-1 scale","Synthetic data privacy metrics get a common scale","Privacy metrics for synthetic data, standardized","One scale to rank all 17 privacy metrics","All 17 synthetic-data privacy metrics on one scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The catalog's usefulness depends on the assumption that rescaling a published metric by a monotone transform (for example, $1-\\mathrm{DVP}$ or $1-\\sigma(\\log \\mathrm{DCR})$) preserves the metric's meaning well enough that scores from different metrics can be compared on the same scale across datasets and attribute types.","fun_headline_variants_meta":{"raw":{"variants":["17 privacy metrics unified on a 0-to-1 scale","Synthetic data privacy metrics get a common scale","Privacy metrics for synthetic data, standardized","One scale to rank all 17 privacy metrics","All 17 synthetic-data privacy metrics on one scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":3966,"prompt_tokens":881,"completion_tokens":3085,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":3012}},"tokens_in":497,"tokens_out":3085,"duration_ms":24050,"temperature":1.0,"reasoning_tokens":3012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:10:50.495448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute both the raw and rescaled versions of DVP on a suite of synthetic datasets with known re-identification rates; if $1-\\mathrm{DVP}$ does not track the fraction of real records an adversary can actually re-identify, the rescaling is an artifact rather than a transparency fix.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ZCAP and GCAP, the categorical attribute-disclosure baselines whose match-and-correct-match structure the review formalizes."},{"cited_title":"Mooney, and Bradley A","cited_arxiv_id":null,"evidence_quote":"Supplies the Attribute Inference Risk (AIR) metric, including its F1 scoring and entropy-based tuple weighting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces NNDR, DCR, and HiddenRate, the metrics that use lower-dimensional embeddings and the rescaling with the sigmoid of log DCR."}],"review_version":1}