Pith. sign in

REVIEW 3 major objections 5 minor 48 references

A holdout control group separates genuine privacy disclosures in synthetic data from phantom coincidences, yielding formal leakage bounds without model access.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:06 UTC pith:R4UO3L5X

load-bearing objection Useful packaged audit for synthetic data, but the extractor-relative epsilon bounds and the unsupported compute claim need fixing before the central claims are credible. the 3 major comments →

arxiv 2606.16952 v2 pith:R4UO3L5X submitted 2026-06-15 cs.LG cs.AIstat.APstat.MEstat.ML

Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data

Kareem Amin , Rudrajit Das , Alessandro Epasto , Adel Javanmard , Dennis Kraft , M\'onica Ribero , Sergei Vassilvitskii This is my paper
classification cs.LG cs.AIstat.APstat.MEstat.ML
keywords synthetic data privacymembership inferencedifferential privacy auditingphantom disclosuresholdout controldisclosure detectionLLM memorizationempirical privacy bounds
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when auditing synthetic data for privacy leakage, many apparent disclosures are 'phantom' matches that would occur even if a user's record were never used to train the generator. It establishes a statistical framework that randomly splits the source corpus into training and holdout sets, counts rare features from each that appear in the synthetic output, and tests whether the training-set match rate exceeds the holdout baseline. If it does, the excess is attributable to training-set membership, providing a formal lower bound on the privacy parameter epsilon and a membership-inference attack. The framework runs with no access to the generative model, no inserted canary records, and no shadow-model training — only synthetic output and a held-out control sample. If correct, it gives data owners a practical, explainable audit that surfaces which private features actually leaked, and it warns that naive disclosure counts overstate risk when phantoms are ignored.

Core claim

The paper's central claim is that a privacy disclosure in synthetic data only counts as evidence of leakage if it occurs more often against the training corpus than against a held-out corpus drawn from the same distribution. It formalizes this by defining disclosures as rare features that appear in both source and synthetic data, then testing the null hypothesis that the generating mechanism learns nothing from training records (zero learning) or that it satisfies epsilon-differential privacy. Under these nulls, the training/holdout assignment is random, so the number of training-side feature matches follows a known distribution; the paper derives critical values and, when they are exceeded,

What carries the argument

The Feature Match Test is the engine: for a disclosure class (PII strings, n-grams, or embedding similarities), each rare feature that appears in the synthetic output records which source users possess it; the test statistic T counts how many possession events land in the training set. Because each record entered training independently with probability p, T under the null is a sum of independent Bernoulli-weighted counts, so Hoeffding bounds give critical values c_alpha and c_{eps,alpha}. The DP test replaces the Bernoulli mean p by q_eps = p/(p+(1-p)e^{-eps}), and Theorem 2 inverts the observed T into a lower bound on eps. This decomposition separates true disclosures from phantoms and powe

Load-bearing premise

The audit's phantom baseline requires a clean, randomly partitioned holdout sample from the same underlying distribution as the training data; if the synthetic data was generated from the full corpus before an audit was planned, the controls and both hypothesis tests cannot be run.

What would settle it

Reject the zero-learning null at a rate far above the nominal significance level using a generator that is provably independent of the training data (e.g., random sampling from a fixed public corpus) across many random train/holdout splits; this would indicate the test's type-I error is not controlled. Alternatively, draw the holdout from a slightly different distribution (e.g., a different time period) and observe a false positive, demonstrating the protocol's dependence on the holdout assumption.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Auditors can obtain empirical lower bounds on privacy leakage that are tighter than prior data-based methods, from a single synthetic release and a holdout set, with no access to the generative model.
  • Raw disclosure counts overstate privacy risk: the phantom-disclosure rate must be measured on holdout data, and in several experiments phantoms account for more than a third of apparent matches.
  • For differentially private (DP-SGD) generators, observed matches are statistically indistinguishable from phantoms, supporting the efficacy of formal privacy protections.
  • Semantic or embedding-based matching detects leakage that verbatim n-gram checks miss, revealing rewritten records that retain the meaning of private training examples.
  • The framework produces explainable witnesses of leakage — specific leaked features and affected records — rather than a single opaque membership score.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The audit's sensitivity depends on the chosen feature extractor; a deployment that passes with a weak extractor (e.g., PII only) might still leak semantically, so a natural extension is to require a battery of extractors or adaptive feature search and report the sensitivity of the conclusions.
  • The phantom rate itself is informative: a generator whose outputs are distributionally close enough to reproduce holdout PII already merits caution even if no true disclosures are found, since the same property could amplify real leakage once the data is private.
  • The controlled-experiment framing could generalize beyond synthetic generation to any data-processing pipeline where one can partition source data and observe outputs, such as retrieval or summarization systems.
  • When no clean holdout exists because the synthetic data was already generated from the full corpus, the true-versus-phantom distinction collapses; a substitute could use an independent public corpus from the same domain, but that would only be a weak proxy for the required counterfactual.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an empirical auditing framework for synthetic data that, given a randomized train/holdout split, a generative mechanism, and a user-selected feature extractor (PII, n-grams, or embeddings), counts matches between rare features in the training data and the synthetic output, compares them to matches on the holdout ('phantoms'), and tests two null hypotheses: zero learning and epsilon-DP. Two tests are introduced, a feature-match test and a user-match membership inference attack, with formal type I error control (Theorems 1 and 3) and empirical epsilon lower bounds (Theorem 2). Experiments on rewrites, SFT, and DP-SGD show significant leakage for non-DP methods, phantom-dominated matches for DP-SGD, and favorable comparison against the canary-echo baseline of Meeus et al. The framework requires no model access, no canaries, and no shadow models, but it does require a controlled train/holdout partition and produces extractor-relative conclusions.

Significance. If the results hold, the contribution is significant. The phantom-disclosure baseline is a genuinely useful and often-overlooked accounting device: the paper quantifies it (e.g., 271 of 763 Finance PII matches are phantoms) and shows that raw disclosure counts overstate risk. The statistical tests have formal type I error control (Theorems 1-3) with self-contained proofs in Appendix C, generalizing the posterior-bound argument of Steinke et al. to feature-incidence matrices and arbitrary sampling probabilities. The method is model-agnostic, avoids canary design and reference-model training, and the experimental sweep across rewrites, SFT, and DP-SGD gives a clean qualitative picture: non-DP methods leak, while DP-SGD matches are indistinguishable from phantoms. These strengths justify publication provided the overclaims and extractor-dependence issues identified below are addressed; the statistical skeleton itself appears sound.

major comments (3)
  1. [§1, §7, Table 3, Theorem 2] The abstract's unqualified claim of 'empirical lower bounds on privacy leakage' is contradicted by the extractor-relative nature of the audit. The bounds in Theorem 2 are computed on the chosen extractor's matches, and Table 3 shows the dependence is not second-order: Finance rewrites yield epsilon=0.0 with DLP but 2.04 with strings; NYT yields epsilon=5.28 with embeddings but 0.0 with DLP and 0.65 with strings. With only a weak extractor (DLP on Finance), the test fails to reject the zero-learning null despite substantial verbatim leakage. Section 7 acknowledges this, but the abstract and conclusion do not. The paper needs either (a) an explicit, always-present qualification that the audit is conditional on the feature set, along with guidance on choosing a sufficient set of extractors or a combined test with multiplicity control, or (b) a demonstration that some extractor-agnostic conc
  2. [§1, §6.2, Table 6 caption] The abstract and introduction claim the framework 'requires orders of magnitude fewer computational resources' than shadow-model/canary alternatives. The only quantitative evidence in the paper is the Table 6 caption: 'requiring 5× fewer computational resources for training and sampling.' 5× is not an order of magnitude. Either the claim should be corrected to 'fewer' (or 'substantially fewer'), or a benchmark demonstrating an order-of-magnitude gap should be provided. This overstatement is not central to the statistical contribution, but it is one of the paper's stated advantages and should be fixed.
  3. [§6.2, after Table 3] The sentence 'our method using either embeddings or strings always outperforms the Meeus et al. baseline' is contradicted by Table 3. Examples: Postings strings epsilon=0.02 vs baseline 0.24; Finance embeddings epsilon=0.25 vs baseline 1.77; Panorama+ DLP/strings epsilon=0.0 vs baseline 0.39. The claim is only sustainable if the best extractor is selected per dataset, which is not a fair comparison to a fixed baseline. The text should report the comparison honestly (e.g., best-per-dataset extractor, with the caveat that extractor choice matters), and the abstract's 'tighter than prior data-based auditing methods' should be qualified accordingly.
minor comments (5)
  1. [§5.1.1] The sentence 'The test also instantiates a p-value, p_val = Pr(T≥c_alpha)' is not a definition of a p-value; the correct p-value is given in Lemma 2. The notation should be corrected to avoid confusion between the critical value, the type I error, and the observed-test-statistic p-value.
  2. [Table 1] The rows for NYT comments (DLP) and Tweets (DLP) show '35' and '68' with dashes for disclosure counts. It is unclear whether these are |F1| values with zero synthetic matches, or formatting artifacts. A footnote or explicit '0' would help.
  3. [§6.1] The extractor label 'DLP (Semantic PII)' is misleading: the Google Cloud DLP tool detects PII by patterns and dictionaries, not by semantics. Rename to 'DLP (PII detection)' to avoid conflating with the embedding-based semantic extractor.
  4. [§6.3] The text 'all DP-SGD estimates are zero' is imprecise. The experiments fail to reject the zero-learning null for DP-SGD, so the lower bounds are not significantly positive; the paper should say 'not significant' rather than 'zero' to avoid implying exact zero leakage.
  5. [§4, §7] The core requirement of a controlled train/holdout partition is stated in the conclusion but should appear prominently in the introduction, since it determines whether the audit can be applied post-hoc or only in pre-planned deployments.

Circularity Check

0 steps flagged

Core statistical derivation is self-contained: epsilon lower bounds are confidence bounds from the null distribution, not fitted predictions.

full rationale

The central statistical derivation is self-contained. The zero-learning null is a well-defined counterfactual (Eq. 3 conditions on the observed feature-incidence matrix and randomizes training indicators; no leakage assumption is baked in). The epsilon lower bounds in Theorem 2 are genuine lower confidence bounds obtained by inverting Hoeffding's inequality under the epsilon-DP null, via stochastic domination (Proposition 1), whose proof follows Steinke et al. [40], an independent external result. No parameter is fitted and then reported as a prediction: the bound is a monotone transformation of the observed test statistic with calibrated 1-alpha coverage. The phantom baseline is a true holdout control. The only self-citation ([4]) is contextual in related work, not load-bearing. Limitations about requiring a controlled partition and extractor-relative bounds are explicit scope conditions, not circular admissions. Table 3 extractor variation is a false-negative risk, not a construction circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The framework's free parameters are audit-design choices (split probability, rarity thresholds, extractor bounds), not quantities fitted to the target result. The core assumptions are the random partition and the ε-DP null, plus standard concentration inequalities. No hidden entity is introduced; 'phantom disclosure' is an operational definition with direct holdout evidence.

free parameters (6)
  • sampling probability p (audit split) = 0.5 (experiments)
    Chosen by auditor; defines the null distribution of T; the test is calibrated to this p.
  • rarity parameter k (feature frequency) = 1 (main tables)
    Defines F^k as features appearing exactly k times; affects counts and power; varied in appendix.
  • substring length bounds n_min, n_max = 11, 20
    Feature extraction hyperparameters for the regurgitation class.
  • embedding rarity quantile q = 0.05
    Threshold for semantic-reconstruction rareness; changes two-sample test power.
  • nearest-neighbor count κ (embeddings) = 10
    Used for embedding similarity ranking.
  • significance level α = 0.05
    Standard; controls type I error.
axioms (4)
  • domain assumption Records are sampled into training independently with probability p (Section 4).
    Random partition is what makes the test causal; required for the phantom baseline.
  • domain assumption Under the DP null, M is ε-DP with respect to the training set (used in Theorem 1/3 proofs).
    The posterior bounds in Proposition 1 depend on ε-DP.
  • standard math Hoeffding's inequality, Bayes' rule, stochastic domination lemmas (Appendices B/C).
    Standard tools used in proofs.
  • domain assumption The feature extractor and rarity functions are fixed before the audit and characterize the disclosure threats.
    The audit only sees what the extractor surfaces.
invented entities (1)
  • Phantom disclosures independent evidence
    purpose: Baseline concept: rare features that match synthetic output but belong to holdout (non-training) records, quantifying coincidence/generalization.
    Defined operationally via holdout matches; measured empirically (e.g., 271/763 in Finance SFT DLP). Not an unobserved entity.

pith-pipeline@v1.3.0-alltime-deepseek · 24381 in / 24323 out tokens · 231642 ms · 2026-08-02T11:06:50.359644+00:00 · methodology

0 comments
read the original abstract

The rapid adoption of generative AI and Large Language Models (LLMs) has spurred interest in synthetic data as a privacy-preserving alternative to sensitive real-world datasets. However, generating high-utility synthetic data often carries the risk of memorizing and regurgitating private information from the training corpus. In this work, we present a customizable empirical auditing framework designed to detect and explain such data disclosures. Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures''-where the system incidentally generates a user's data. By partitioning input data into training and holdout sets and applying rigorous statistical hypothesis testing, we determine if observed disclosures are consistent with strict privacy baselines, such as zero-learning or specific Differential Privacy (DP) bounds. Crucially, this approach requires no model access, no canary insertion, and no reference model training -only the synthetic output and a held-out control set. We demonstrate that this framework effectively functions as a membership inference attack, providing empirical lower bounds on privacy leakage that are tighter than prior data-based auditing methods. Our approach is model-agnostic, applies to any synthetic data generation mechanism, and requires orders of magnitude fewer computational resources than shadow-model or canary-based alternatives.

Figures

Figures reproduced from arXiv: 2606.16952 by Adel Javanmard, Alessandro Epasto, Dennis Kraft, Kareem Amin, M\'onica Ribero, Rudrajit Das, Sergei Vassilvitskii.

Figure 1
Figure 1. Figure 1: Overview of the audit pipeline. Data consists of training and held-out sets. Training data is processed by a synthetic generation algorithm to produce synthetic data. All three datasets – training, held-out, and synthetic – are processed for features. Rare features appearing in held￾out and synthetic data are phantom disclosures. Rare features in training and synthetic are true disclosures. We now have all… view at source ↗
Figure 2
Figure 2. Figure 2: Impact of feature rarity (k) on the audit lower bound pˆ. As k increases (features shared by more records), the number of valid audit features |Fk| decreases. This reduction in sample size leads to looser statistical bounds, reducing the power of the tests. 2 4 6 8 10 Number of records with feature overall 0.0 0.2 0.4 0.6 0.8 Test statistic and CI lower bound. Null: p=0.5 DLP 2 4 6 8 10 Number of records w… view at source ↗
Figure 3
Figure 3. Figure 3: Point estimates and confidence intervals for varying k. The solid line represents the empirical match rate (point estimate). The shaded region illustrates the gap between the point estimate and the conservative lower bound ˆp used for auditing. The gap highlights the increased statistical uncertainty caused by the lack of non-unique features in the synthetic corpus at higher frequencies. 26 [PITH_FULL_IMA… view at source ↗
Figure 4
Figure 4. Figure 4: Two-sample test results comparing the distribution of cosine similarities between training [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two-sample test comparing distributions of training-to-synthetic and holdout-to [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 15 linked inside Pith

  1. [1]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308–318, 2016

    2016

  2. [2]

    The us census bureau adopts differential privacy

    John M Abowd. The us census bureau adopts differential privacy. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2867–2867, 2018

    2018

  3. [3]

    Anonymous-by-construction: An llm-driven framework for privacy-preserving text.arXiv preprint arXiv:2603.17217, 2026

    Federico Albanese, Pablo Ronco, and Nicol´ as D’Ippolito. Anonymous-by-construction: An llm-driven framework for privacy-preserving text.arXiv preprint arXiv:2603.17217, 2026

    arXiv 2026

  4. [4]

    Private prediction for large-scale synthetic text gener- ation

    Kareem Amin, Alex Bie, Wei Kong, Alexey Kurakin, Natalia Ponomareva, Umar Syed, An- dreas Terzis, and Sergei Vassilvitskii. Private prediction for large-scale synthetic text gener- ation. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7244–7262, 2024

    2024

  5. [5]

    Reconstructing training data with informed adversaries

    Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In43rd IEEE Symposium on Security and Privacy (SP), pages 1138–1156. IEEE, 2022

    2022

  6. [6]

    Optimizing canaries for privacy auditing with metagradient descent.arXiv preprint arXiv:2507.15836, 2025

    Matteo Boglioni, Terrance Liu, Andrew Ilyas, and Zhiwei Steven Wu. Optimizing canaries for privacy auditing with metagradient descent.arXiv preprint arXiv:2507.15836, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tram` er. Membership inference attacks from first principles. In2022 IEEE Symposium on Security and Privacy (S&P), pages 1897–1914. IEEE, 2022

    1914

  8. [8]

    Quantifying memorization across neural language models, 2023.URL https://arxiv

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023.URL https://arxiv. org/abs/2202.07646, 2202, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    The secret sharer: Evaluating and testing unintended memorization in neural networks

    Nicholas Carlini, Chang Liu, ´Ulfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In28th USENIX security symposium (USENIX security 19), pages 267–284, 2019

    2019

  10. [10]

    Gan-leaks: A taxonomy of membership inference attacks against generative models

    Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. InACM SIGSAC Conference on Computer and Communications Security, 2020

    2020

  11. [11]

    Calibrating noise to sensi- tivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensi- tivity in private data analysis. InTheory of cryptography conference, pages 265–284. Springer, 2006

    2006

  12. [12]

    Calibrating noise to sen- sitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. InTheory of Cryptography Conference (TCC), 2006

    2006

  13. [13]

    Differ- entially private optimization with sparse gradients.Advances in Neural Information Processing Systems, 37:63406–63440, 2024

    Badih Ghazi, Crist´ obal Guzm´ an, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Differ- entially private optimization with sparse gradients.Advances in Neural Information Processing Systems, 37:63406–63440, 2024

    2024

  14. [14]

    Property testing for differential privacy, 2019

    Anna Gilbert and Audra McMillan. Property testing for differential privacy, 2019. 21

    2019

  15. [15]

    Cloud Data Loss Prevention (Cloud DLP) api.https://docs.cloud.google.com/ sensitive-data-protection/docs/infotypes-reference, 2026

    Google. Cloud Data Loss Prevention (Cloud DLP) api.https://docs.cloud.google.com/ sensitive-data-protection/docs/infotypes-reference, 2026. Accessed: 2026-02-07

    2026

  16. [16]

    Bounding train- ing data reconstruction in private (deep) learning

    Chuan Guo, Brian Karrer, Kamalika Chaudhuri, and Laurens van der Maaten. Bounding train- ing data reconstruction in private (deep) learning. InInternational Conference on Machine Learning (ICML), 2022

    2022

  17. [17]

    The surprising effectiveness of membership inference with simple n-gram coverage.arXiv preprint arXiv:2508.09603, 2025

    Skyler Hallinan, Jaehun Jung, Melanie Sclar, Ximing Lu, Abhilasha Ravichander, Sahana Ramnath, Yejin Choi, Sai Praneeth Karimireddy, Niloofar Mireshghallah, and Xiang Ren. The surprising effectiveness of membership inference with simple n-gram coverage.arXiv preprint arXiv:2508.09603, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Reconstruction and membership inference attacks against generative models.arXiv preprint arXiv:1906.03006, 2019

    Benjamin Hilprecht, Martin H¨ arterich, and Daniel Bernau. Reconstruction and membership inference attacks against generative models.arXiv preprint arXiv:1906.03006, 2019

    Pith/arXiv arXiv 1906

  19. [19]

    Statistics and causal inference.Journal of the American Statistical Associa- tion, 81(396):945–960, 1986

    Paul W Holland. Statistics and causal inference.Journal of the American Statistical Associa- tion, 81(396):945–960, 1986

    1986

  20. [20]

    Auditing differentially private machine learning: How private is private SGD?Advances in Neural Information Processing Systems, 33:22205–22216, 2020

    Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD?Advances in Neural Information Processing Systems, 33:22205–22216, 2020

    2020

  21. [21]

    Evaluating differentially private machine learning in practice

    Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. InUSENIX Security Symposium, 2019

    2019

  22. [22]

    Synthetic data–what, why and how? arXiv preprint arXiv:2205.03257, 2022

    James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data–what, why and how? arXiv preprint arXiv:2205.03257, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    The enron corpus: A new dataset for email classification research

    Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. InEuropean conference on machine learning, pages 217–226. Springer, 2004

    2004

  24. [24]

    Linkedin job postings (2023 - 2024), 2024

    Arsh Koneru. Linkedin job postings (2023 - 2024), 2024

    2023

  25. [25]

    Harnessing large-language models to generate private synthetic text.arXiv preprint arXiv:2306.01684, 2023

    Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing large-language models to generate private synthetic text.arXiv preprint arXiv:2306.01684, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Differentially private synthetic data via apis 3: Using simulators instead of foundation model.arXiv preprint arXiv:2502.05505, 2025

    Zinan Lin, Tadas Baltrusaitis, Wenyu Wang, and Sergey Yekhanin. Differentially private synthetic data via apis 3: Using simulators instead of foundation model.arXiv preprint arXiv:2502.05505, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Dif- ferentially private synthetic data via foundation model apis 1: Images.arXiv preprint arXiv:2305.15560, 2023

    Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. Dif- ferentially private synthetic data via foundation model apis 1: Images.arXiv preprint arXiv:2305.15560, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Differentially private language models for secure data sharing

    Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differentially private language models for secure data sharing. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), pages 4860–4873. ACL, 2022

    2022

  29. [29]

    Mann-whitney u test.The Corsini encyclopedia of psychology, pages 1–1, 2010

    Patrick E McKnight and Julius Najab. Mann-whitney u test.The Corsini encyclopedia of psychology, pages 1–1, 2010. 22

    2010

  30. [30]

    Achilles’ heels: vulnerable record identification in synthetic data publishing

    Matthieu Meeus, Florent Guepin, Ana-Maria Cret ¸u, and Yves-Alexandre de Montjoye. Achilles’ heels: vulnerable record identification in synthetic data publishing. InEuropean Symposium on Research in Computer Security, pages 380–399. Springer, 2023

    2023

  31. [31]

    Copyright traps for large language models.arXiv preprint arXiv:2402.09363, 2024

    Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre De Montjoye. Copyright traps for large language models.arXiv preprint arXiv:2402.09363, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    The canary’s echo: Auditing privacy risks of llm-generated synthetic text.arXiv preprint arXiv:2502.14921, 2025

    Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-B´ eguelin, Shruti Tople, and Reza Shokri. The canary’s echo: Auditing privacy risks of llm-generated synthetic text.arXiv preprint arXiv:2502.14921, 2025

    arXiv 2025

  33. [33]

    Tight auditing of differentially private machine learning

    Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tram` er, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. In32nd USENIX Security Symposium (USENIX Security 23), pages 1631–1648, 2023

    2023

  34. [34]

    How to dp-fy your data: A practical guide to generating synthetic data with differential privacy

    Natalia Ponomareva, Zheng Xu, H Brendan McMahan, Peter Kairouz, Lucas Rosenblatt, Vincent Cohen-Addad, Crist´ obal Guzm´ an, Ryan McKenna, Galen Andrew, Alex Bie, et al. How to dp-fy your data: A practical guide to generating synthetic data with differential privacy. arXiv preprint arXiv:2512.03238, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon.arXiv preprint arXiv:2406.17746, 2024

    USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir SV, Mohammad Aflah Khan, Jay- deep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, et al. Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon.arXiv preprint arXiv:2406.17746, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Differential privacy defenses and sampling attacks for membership inference

    Shadi Rahimian, Tribhuvanesh Orekondy, and Mario Fritz. Differential privacy defenses and sampling attacks for membership inference. InACM Workshop on Artificial Intelligence and Security, 2021

    2021

  37. [37]

    Panorama: A synthetic pii-laced dataset for studying sensitive data memorization in llms.arXiv preprint arXiv:2505.12238, 2025

    Sriram Selvam and Anneswa Ghosh. Panorama: A synthetic pii-laced dataset for studying sensitive data memorization in llms.arXiv preprint arXiv:2505.12238, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789, 2023

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017

    2017

  40. [40]

    Privacy auditing with one (1) training run.Advances in Neural Information Processing Systems, 36:49268–49280, 2023

    Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run.Advances in Neural Information Processing Systems, 36:49268–49280, 2023

    2023

  41. [41]

    Privacy-preserving in-context learning with differentially private few-shot generation

    Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zi- nan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially private few-shot generation. In12th International Conference on Learning Representations (ICLR 2024), 2024

    2024

  42. [42]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi` ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024. 23

    Pith/arXiv arXiv 2024

  43. [43]

    Proving membership in llm pretraining data via data watermarks.arXiv e-prints, pages arXiv–2402, 2024

    Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. Proving membership in llm pretraining data via data watermarks.arXiv e-prints, pages arXiv–2402, 2024

    2024

  44. [44]

    Synthetic-PII-Financial-Documents-North-America: A synthetic dataset for training language models to label and detect pii in domain specific formats, June 2024

    Alex Watson, Yev Meyer, Maarten Van Segbroeck, Matthew Grossman, Sami Torbey, Piotr Mlocek, and Johnny Greco. Synthetic-PII-Financial-Documents-North-America: A synthetic dataset for training language models to label and detect pii in domain specific formats, June 2024

    2024

  45. [45]

    Differentially pri- vate synthetic data via foundation model apis 2: Text

    Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haoming Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, and Sergey Yekhanin. Differentially pri- vate synthetic data via foundation model apis 2: Text. InProceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235, pages 54531–54560. PMLR, 2024

    2024

  46. [46]

    Enhanced membership inference attacks against machine learning models

    Jiayuan Ye, Aadithya Maddi, Sumanth K Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. InPro- ceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS 2022), pages 3093–3106. ACM, 2022

    2022

  47. [47]

    Synthetic text generation with differential privacy: A simple and practical recipe

    Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, James McAnallen, Huseyin Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1321–1342. ACL, 2023

    2023

  48. [48]

    A form capturing personal, financial, and employment details of a loan applicant, including loan amount, purpose, and supporting documents

    Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks.arXiv preprint arXiv:2312.03262, 2023. 24 A Experimental details A.1 Datasets We leverage three existing publicly available datasets for synthetic data generation. FinanceThis dataset contains 55,940 fictitious financial records generated using the Gretel pa...

    Pith/arXiv arXiv 2023