A Review of Privacy Metrics for Privacy-Preserving Synthetic Data Generation

Daniele Dell'Aglio; Frederik Marinus Trudslev; Juan Manuel Rodriguez; Martin B{\o}gsted; Matteo Lissandrini

REVIEW 6 major objections 5 minor 11 references

A Review of Privacy Metrics for Privacy-Preserving Synthetic Data Generation

T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that the 17 privacy metrics used to evaluate synthetic data can be restated under a single definition, with explicit attack assumptions and rescaling to a common [0,1] privacy-risk scale.

desk verdict A useful catalog idea undermined by a cluster of formula errors, including an inverted weighting scheme in the ID score; fixable, but not trustworthy as submitted. read the letter →

arxiv 2507.11324 v2 pith:6PITQ5BZ submitted 2025-07-15 cs.CR cs.DB

Frederik Marinus Trudslev , Matteo Lissandrini , Juan Manuel Rodriguez , Martin B{o}gsted , Daniele Dell'Aglio This is my paper

classification cs.CRcs.DB

keywords privacy-preservingsyntheticdataprivacymetricsdifferentialre-identificationriskmembershipinferenceattributedisclosureevaluationscale

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Privacy-preserving synthetic data generation aims to release synthetic datasets that hide individuals, but the standard guarantee, differential privacy, gives a budget $\varepsilon$ that is hard for non-experts to interpret. This paper argues that the many privacy metrics proposed to make this risk concrete can be presented as a single coherent catalog: 17 metrics, each defined by explicit assumptions about the adversary and by a mathematical formula. To make them comparable, the paper fixes a common scale on which 0 means complete privacy and 1 means no privacy, and it rescales several published metrics that originally pointed in the opposite direction or had unbounded range. If the catalog is right, practitioners can choose a metric by matching the attack it models and read its output directly as a privacy-risk score.

What carries the argument

The load-bearing object is Definition 2, the unified privacy metric $p: \mathcal{D}\times\mathcal{D}\to[0,1]$ with 0 = complete privacy and 1 = no privacy, together with the two-component computation pattern: simulation-based metrics build a guess relation $G\subset Y\times Z$ and an oracle-corrected relation $C\subseteq R$, then score the attack; distance-based metrics skip the adversary and compute a score directly from nearest-neighbour distances. The paper's rescaling identities — e.g., $1-\mathrm{DVP}$, $1-\mathrm{Auth}$, $1-\mathrm{NNAA}$, $1-\sigma(\log \mathrm{DCR})$, and $\sigma(\mathrm{MDCR})$ — are the mechanisms that force the published metrics onto the common scale.

What would settle it

Compute both the raw and rescaled versions of DVP on a suite of synthetic datasets with known re-identification rates; if $1-\mathrm{DVP}$ does not track the fraction of real records an adversary can actually re-identify, the rescaling is an artifact rather than a transparency fix.

Watch

Extended reading notes

Core claim

The paper's central claim is that 17 published privacy metrics used in privacy-preserving synthetic data generation can be stated on a common formal footing: each is a function $p(Y,Z)$ from a real dataset and a synthetic dataset to $[0,1]$, with 0 meaning complete privacy and 1 meaning no privacy. To achieve this, the paper restates the metrics in two families — simulation-based, where an adversary's guessed matches $G$ are checked against correct matches $C$ by an oracle, and distance-based, where distances between real and synthetic records are analysed directly — and it supplies rescaled versions of metrics whose original direction or range did not fit, such as $\mathrm{DVP}' = 1-\mathrm{DVP}$, $\mathrm{Auth}'=1-\mathrm{Auth}$, and $\mathrm{DCR}' = 1-\sigma(\log \mathrm{DCR})$. The result is a reference catalog that pairs each metric with the attack assumptions it embodies.

Load-bearing premise

The catalog's usefulness depends on the assumption that rescaling a published metric by a monotone transform (for example, $1-\mathrm{DVP}$ or $1-\sigma(\log \mathrm{DCR})$) preserves the metric's meaning well enough that scores from different metrics can be compared on the same scale across datasets and attribute types.

Editorial extensions

If this is right

A practitioner can now look up any of the 17 metrics, see the attack it assumes, and read its score as a privacy risk between 0 and 1 without re-deriving the formula.
The explicit assumptions make mismatches visible: for example, ZCAP and GCAP only apply to categorical sensitive attributes, while AIR is the variant that adds continuous attributes with a 10% tolerance.
Rescaled metrics such as $\mathrm{DVP}'=1-\mathrm{DVP}$ and $\mathrm{Auth}'=1-\mathrm{Auth}$ let metrics that originally measured distance or authenticity be compared in the same direction as the rest of the catalog.
The division into simulation-based and distance-based metrics gives a shared vocabulary for discussing which privacy attack a score is actually quantifying.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper does not take is to check whether the 17 rescaled scores actually move together on benchmark datasets when the privacy risk changes; if they do not, the common scale is formal rather than behavioral.
Because the rescaling maps include nonlinear ones like $1-\sigma(\log \mathrm{DCR})$, two datasets can be ordered one way by the raw DCR and the opposite way by $\mathrm{DCR}'$; whether this flips practical privacy decisions is an open empirical question.
The taxonomy suggests a selection rule the paper leaves implicit: simulation-based metrics suit threat models with a concrete adversary, while distance-based metrics suit quick audits without a specified adversary.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Main takeaway: the paper is a catalog of 17 privacy metrics with the goal of making their mathematical formulations precise enough for practitioners to use. That is a genuinely useful goal, and the structure is sensible: each metric gets a threat model, an equation, and a rescaling to a common 0-1 privacy-risk scale. The survey coverage is good, and the authors clearly know the synthetic-data literature. The companion PrivEval tool is a nice touch and gives readers somewhere to check the implementation.

Unfortunately, the formulas are not reliable, and since the abstract explicitly promises 'correct assumptions and mathematical formulations,' that is a load-bearing problem. The stress-test note is right: in Section 2.2.5, the weight w(a,v) = 1/(H+ε) is defined so low-entropy attributes get high weight, but Eq. (19) divides by that weight, so high-entropy attributes end up amplified and low-entropy ones shrunk—exactly the opposite of the stated rationale. That is not a typo in a rare metric; it affects every ID distance in Eq. (20).

The other errors the reader flagged are real too. Eq. (30) defines h(a) as max minus max, which is identically zero. Eq. (6) is a malformed F1 expression that does not reduce to the standard formula. Eq. (24) has a variable mistake in the argmin (min over y in Z). These are exactly the places where a reference catalog has to be perfect. Also worth noting but less severe: the rescaling choices like DVP' = 1-DVP and Auth' = 1-Auth are monotone flips, which are fine for direction but do not by themselves guarantee comparability across datasets; the paper could say more about that. The fixed thresholds (like CVP's 0.2) appear without justification, but that is inherited from the source papers.

On the positive side, the threat-model taxonomy and the figures are clear, Definition 2 is reasonable, and the authors are honest about which rescalings they introduced. Nothing here looks like bad faith; it reads like a draft that was not checked equation-by-equation.

Who is this for? Practitioners who want to compare metric behavior on their own data and researchers building evaluation frameworks. They should use the original papers and the tool, not these equations, until the errors are fixed.

My call: the catalog deserves a serious referee and a major revision, not a desk rejection. With a careful pass over every equation and a check against the tool's implementation, it could become a genuinely useful reference. As submitted, I would not cite it.

Referee Report

6 major / 5 minor

Summary. The paper proposes to be a reference catalog of privacy metrics for privacy-preserving synthetic data generation (PP-SDG). It defines a common setup (real data Y, synthetic data Z), imposes a unified [0,1] privacy-risk scale in which 0 means complete privacy and 1 means no privacy, and then presents assumptions and mathematical formulations for 17 metrics, divided into simulation-based and distance-based families. Many of the metrics are rescaled or transformed from their original definitions to fit the unified scale, and the authors link the catalog to their PrivEval implementation.

Significance. If the formulations were correct, the paper would fill a useful gap: practitioners could compare privacy metrics on a common scale and would have a single reference for the assumptions behind each metric. The paper also has practical value in making its code available and in being explicit about rescaling choices, which is more than most metric catalogs do. However, the central claim is that the presented formulas are correct, and the manuscript contains several load-bearing formula errors that contradict the accompanying text. Because the paper's product is the mathematical formulation itself, these errors directly undermine its main contribution. The paper should not be accepted until every equation is checked against the original sources and the linked implementation; with correction, the catalog could be a useful reference.

major comments (6)

[§2.2.5, Eq. (19)] The Identifiability (ID) score weighting is inverted. Eq. (18) defines w(a,v)=1/(H(a,v)+ε), so low-entropy attributes receive high weight. Eq. (19) then sets ŷ[a]=y[a]/(w(a,y[a])+ε), which is division by the weight and therefore multiplies by H(a,y[a])+ε. This amplifies high-entropy attributes and damps low-entropy attributes, exactly the opposite of the stated rationale in §2.2.5. The sentence "a low weight means a low contribution to the distance measure" is false for the formula as printed, and because Eq. (20) computes all ID distances on these weighted vectors, the metric as formulated measures the opposite weighting scheme from the one the authors describe.
[§2.2.13, Eq. (30)] The Hitting Rate threshold h(a) is identically zero as printed. The equation reads h(a)=max_{y∈Y} y[a] − max_{y∈Y} y[a] / 30, so the numerator is a maximum minus the same maximum, giving h(a)=0 for every attribute. This makes the "similar values" condition in C_HitR an exact-equality condition on continuous attributes, which contradicts the text's description of attribute-dependent thresholds. The formula should presumably be (max_y y[a] − min_y y[a])/30 or a similar range-based expression, but as written it is not a usable threshold.
[§2.1.3, Eq. (6)] The F1 formula for Attribute Inference Risk is malformed. As typeset, the expression does not reduce to the standard F1 = 2TP/(2TP+FP+FN), and the nesting of fractions is not well defined. Since AIR is defined as a weighted average of this F1 value, the quantitative behavior of the entire AIR metric is unclear from the printed formula. This needs to be corrected and verified against the source definition in [8].
[§2.2.8, Eq. (24)] The set C_DCR has an ill-typed argmin. The equation writes z = arg min_{y∈Z} d_E(Φ_k(y), Φ_k(z)), but z is the variable being defined and y is the outer quantifier; the minimization should be over z∈Z for each fixed y. As written, the set cannot be constructed as intended. This is not a mere typo, because DCR is then computed as an average over this set, so the formula does not define a computable metric.
[§2.2.10, Eq. (28)] The NNAA score is defined as a sum of two probabilities rather than their average. The equation sums the fraction of real points closer to a synthetic point than to a real point with the fraction of synthetic points closer to a synthetic point than to a real point. Since each fraction is in [0,1], the sum can exceed 1, and subtracting it from 1 in Eq. (28) can produce negative values. This violates the paper's own Definition 2 and contradicts the text's statement that 0.5 corresponds to an adversary with equal chance of guessing correctly. The intended definition should be the average of the two probabilities.
[§2.2.9, Eq. (27)] The rescaling of MDCR as σ(MDCR) cannot produce values close to 0 for any nonnegative distance ratio. MDCR is a ratio of distances and is therefore nonnegative, and σ(x)≥0.5 for x≥0. The text states that an MDCR' score close to 0 means low re-identification risk, but the smallest possible value is σ(0)=0.5. If the goal is a rescaling that spans the full [0,1] range, a transformation such as σ(log(MDCR)) would be needed, not the one printed in Eq. (27).

minor comments (5)

[§2.2.7, Eq. (23)] In the NNDR formula, the distances d_E(y',z) and d_E(y'',z) should be d_E(Φ_k(y'),Φ_k(z)) and d_E(Φ_k(y''),Φ_k(z)) to be consistent with the mapping defined in Eq. (22). Also, the text refers to "NDRR" where "NNDR" is meant.
[§2.2.8, Eq. (25)] The DCR' transformation uses log(DCR), which is undefined when DCR=0. The paper should either restrict the domain or add a small epsilon inside the logarithm, and should state how zero distances are handled.
[Definition 2, §2.2.6] Definition 2 says that if Z=Y, a well-designed privacy metric should yield 1, but NSND in Eq. (21) gives 0 in that case because all nearest-neighbor distances are zero. The paper should acknowledge that distance-based metrics are not aligned with this expectation, or adjust the framing of Definition 2.
[Throughout] There are several typographical and language issues: "LigtGBM" should be "LightGBM", "neigbour" should be "neighbor", "distinct which real individual" should be "distinguish which real individual", and "a in a smaller vector space" should be "in a smaller vector space".
[§2.1.3, Eq. (6)] The text defining AIR says "quantifies uses the F1 score," which is a grammatical error; it should read "quantifies the attack success using the F1 score."

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the catalog entries are externally cited definitions, and the only self-citation (PrivEval [6]) is contextual rather than load-bearing.

full rationale

The paper's central claim is to state assumptions and mathematical formulations of 17 privacy metrics; the formulations are taken from external sources [2,3,4,5,8], not derived from PrivEval or from the paper's own Definition 2. The rescaling operations (DVP' = 1 - DVP, Auth' = 1 - Auth, NNAA' = 1 - NNAA, DCR' = 1 - sigmoid(log DCR), MDCR' = sigmoid(MDCR)) are explicitly labeled as rescaling to fit the [0,1] convention, e.g., 'This does not fit our definition of a privacy metric in 2, instead one can calculate it as...'; they are monotone relabelings of external metrics, not fitted parameters passed off as predictions. The self-citation to PrivEval [6] identifies the metric selection context but is not used to justify any mathematical formulation. There is therefore no circular reduction of a derived result to an input. The internal inconsistencies noted in the ID score and Hitting Rate definitions are correctness defects in the catalog, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's own contribution rests mainly on a high-level definition (Definition 2) and on rescaling transformations that assume comparability; the ledger lists the arbitrary constants and the structural assumptions behind that framing. No new entities are invented.

free parameters (2)

epsilon (const) = unspecified small constant
Added to denominators in Eqs. 18 and 19 to avoid division by zero; its value changes ID scores substantially for low-entropy attributes, and no value is given.
hard thresholds in CVP, DVP, AIR, HitR = 0.2, 0.8, 10%, 30
Distance thresholds and tolerance values are hard-coded constants inherited from the cited metrics; they determine what counts as identifiable and are not fitted or justified in this paper.

assumptions (4)

domain assumption A privacy metric is bounded in [0,1] and direction 0 = complete privacy, 1 = no privacy
Definition 2 imposes this framing; several original metrics have the opposite direction or are unbounded, and the paper rescales them without justification.
domain assumption Distance metrics computed on raw attribute values are meaningful privacy risk measures without scaling or type-aware preprocessing
Metrics like CVP, NSND, and ID apply Euclidean distance to mixed categorical and continuous attributes directly (Eqs. 11-21), an unstated modeling choice.
domain assumption Each metric's adversary model is accepted as given
For instance, Hidden Rate assumes every synthetic row was generated from a known real row (Eq. 29), an assumption that does not hold for many generative models; the paper catalogues the assumption without examining it.
standard math Background statistics and ML tools (F1, entropy, AUC, MLP, LightGBM) are used as understood
These are invoked throughout Section 2 without proof, which is standard for a review.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Review of Privacy Metrics for Privacy-Preserving Synthetic Data Generation." pith.science (2026). https://pith.science/paper/6PITQ5BZ

@misc{pith2026250711324,
  author       = {Pith},
  title        = {Pith review of: A Review of Privacy Metrics for Privacy-Preserving Synthetic Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PITQ5BZ}},
  note         = {Machine review of arXiv:2507.11324}
}

abstract

Privacy Preserving Synthetic Data Generation (PP-SDG) has emerged to produce synthetic datasets from personal data while maintaining privacy and utility. Differential privacy (DP) is the property of a PP-SDG mechanism that establishes how protected individuals are when sharing their sensitive data. It is however difficult to interpret the privacy budget ($\varepsilon$) expressed by DP. To make the actual risk associated with the privacy budget more transparent, multiple privacy metrics (PMs) have been proposed to assess the privacy risk of the data. These PMs are utilized in separate studies to assess newly introduced PP-SDG mechanisms. Consequently, these PMs embody the same assumptions as the PP-SDG mechanism they were made to assess. Therefore, a thorough definition of how these are calculated is necessary. In this work, we present the assumptions and mathematical formulations of 17 distinct privacy metrics.

Figures

Figures reproduced from arXiv: 2507.11324 by the authors.

**Figure 2.** Metric computation in distance-based privacy metrics. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages

[8]

Mooney, and Bradley A

Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D. Mooney, and Bradley A. Malin

work page
[1]

Generating Multi-label Discrete Patient Records using Generative Adversarial Networks

Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. 2018. Generating Multi- label Discrete Patient Records using Generative Adversarial Networks. arXiv:1703.06490 [cs.LG] https://arxiv.org/abs/ 1703.06490

work page Pith review arXiv 2018
[2]

Morgan Guillaudeux, Olivia Rousseau, Julien Petot, Zineb Bennis, Charles-Axel Dein, Thomas Goronflot, Nicolas Vince, Sophie Limou, Matilde Karakachoff, Matthieu Wargny, and Pierre-Antoine Gourraud. 2023. Patient-centric synthetic data generation, no reason to risk re-identification in biomedical data analysis. npj Digital Medicine 6, 1 (10 Mar 2023),

work page 2023
[3]

Markus Hittmeir, Rudolf Mayer, and Andreas Ekelhart. 2020. A Baseline for Attribute Disclosure Risk in Synthetic Data. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy(New Orleans, LA, USA)(CODASPY ’20). Association for Computing Machinery, New York, NY, USA, 133–143. https://doi.org/10.1145/3374664.3375722

work page arXiv 2020
[4]

Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp

Anton D. Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp. 2024. Syntheval: a framework for detailed utility and privacy evaluation of tabular synthetic data. Data Mining and Knowledge Discovery 39, 1 (2024). https://doi.org/10.1007/s10618-024-01081-4

doi:10.1007/s10618-024-01081-4 2024
[5]

Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar. 2023. Synthcity: facilitating innovative use cases of synthetic data in different data modalities. https://doi.org/10.48550/ARXIV.2301.07573

arXiv doi:10.48550/arxiv.2301.07573 2023
[6]

Frederik Marinus Trudslev, Matteo Lissandrini, Juan Manuel Rodriguez, Martin Bøgsted, and Daniele Dell’Aglio. 2025. PrivEval: a tool for interactive evaluation of privacy metrics in synthetic data generation. Proceedings of the VLDB Endowment 18, 12 (2025), 5271 – 5274. https://doi.org/10.14778/3750601.3750649

arXiv 2025
[7]

Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P. Bennett. 2020. Generation and evaluation of privacy preserving synthetic health data. Neurocomputing 416 (2020), 244–255. https://doi.org/10.1016/j. neucom.2019.12.136

doi:10.1016/j 2020

Show all 11 references

[9]

Drumright, and Mihaela van der Schaar

Jinsung Yoon, Lydia N. Drumright, and Mihaela van der Schaar. 2020. Anonymization Through Data Synthesis Using Generative Adversarial Networks (ADS-GAN). IEEE journal of biomedical and health informatics 24, 8 (Aug. 2020), 2378–2388. https://doi.org/10.1109/JBHI.2020.2980262

2020
[37]

https://doi.org/10.1038/s41746-023-00771-5

doi
[2022]

Nature Communications 13, 1 (Dec

A Multifaceted benchmarking of synthetic electronic health record generation models. Nature Communications 13, 1 (Dec. 2022). https://doi.org/10.1038/s41467-022-35295-1

2022 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [8]

Mooney, and Bradley A

Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D. Mooney, and Bradley A. Malin

work page

[2] [1]

Generating Multi-label Discrete Patient Records using Generative Adversarial Networks

Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. 2018. Generating Multi- label Discrete Patient Records using Generative Adversarial Networks. arXiv:1703.06490 [cs.LG] https://arxiv.org/abs/ 1703.06490

work page Pith review arXiv 2018

[3] [2]

Morgan Guillaudeux, Olivia Rousseau, Julien Petot, Zineb Bennis, Charles-Axel Dein, Thomas Goronflot, Nicolas Vince, Sophie Limou, Matilde Karakachoff, Matthieu Wargny, and Pierre-Antoine Gourraud. 2023. Patient-centric synthetic data generation, no reason to risk re-identification in biomedical data analysis. npj Digital Medicine 6, 1 (10 Mar 2023),

work page 2023

[4] [3]

Markus Hittmeir, Rudolf Mayer, and Andreas Ekelhart. 2020. A Baseline for Attribute Disclosure Risk in Synthetic Data. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy(New Orleans, LA, USA)(CODASPY ’20). Association for Computing Machinery, New York, NY, USA, 133–143. https://doi.org/10.1145/3374664.3375722

work page arXiv 2020

[5] [4]

Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp

Anton D. Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp. 2024. Syntheval: a framework for detailed utility and privacy evaluation of tabular synthetic data. Data Mining and Knowledge Discovery 39, 1 (2024). https://doi.org/10.1007/s10618-024-01081-4

doi:10.1007/s10618-024-01081-4 2024

[6] [5]

Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar. 2023. Synthcity: facilitating innovative use cases of synthetic data in different data modalities. https://doi.org/10.48550/ARXIV.2301.07573

arXiv doi:10.48550/arxiv.2301.07573 2023

[7] [6]

Frederik Marinus Trudslev, Matteo Lissandrini, Juan Manuel Rodriguez, Martin Bøgsted, and Daniele Dell’Aglio. 2025. PrivEval: a tool for interactive evaluation of privacy metrics in synthetic data generation. Proceedings of the VLDB Endowment 18, 12 (2025), 5271 – 5274. https://doi.org/10.14778/3750601.3750649

arXiv 2025

[8] [7]

Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P. Bennett. 2020. Generation and evaluation of privacy preserving synthetic health data. Neurocomputing 416 (2020), 244–255. https://doi.org/10.1016/j. neucom.2019.12.136

doi:10.1016/j 2020