Pith. sign in

REVIEW 6 major objections 5 minor 94 references

Struct-Bench: A Benchmark for Differentially Private Structured Text Generation

T0 review · 6 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A grammar-based benchmark shows DP text generators can't keep structure and semantics together.

desk verdict A genuinely useful benchmark for DP structured text generation with a clean CFG-based evaluation protocol; the empirical claims need error bars and a fairer model pairing, and the KND proxy deserves validation. read the letter →

arxiv 2509.10696 v1 pith:FZLILACL submitted 2025-09-12 cs.CL cs.LG

Shuaiqi Wang , Vikas Raunak , Arturs Backurs , Victor Reis , Pei Zhou , Sihao Chen , Longqi Yang , Zinan Lin
show 2 more authors
Sergey Yekhanin Giulia Fanti
This is my paper · ORCID
classification cs.CLcs.LG
keywords differentialprivacysyntheticdatagenerationstructuredtextcontext-freegrammarevaluationbenchmarkPrivateEvolutionsemanticdiversityTSTR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Struct-Bench is a benchmark and evaluation protocol for differentially private (DP) synthetic data generated from structured datasets whose fields contain natural language. The authors aim to show that a single quality metric cannot describe such synthetic data, and that current state-of-the-art DP generators cannot capture the dataset's structural constraints without sacrificing semantic performance. The protocol works by having the user encode the dataset's expected shape as a context-free grammar, then scoring the synthetic data on structural compliance, node-level and sample-level attribute matches, semantic precision and recall, and downstream label-prediction accuracy. On seven datasets, including conversation logs, paper reviews, product reviews, and tabular records, the benchmark exposes tradeoffs that simpler metrics hide. The paper also shows how its metric profiles guided modifications to Private Evolution that improve structural validity and semantic diversity at no additional privacy cost.

What carries the argument

The load-bearing mechanism is the user-supplied context-free grammar (CFG) that encodes the dataset's structural schema. Every sample, real or synthetic, is parsed with this grammar into typed nodes; a tree-pattern tool locates designated key-node pairs, and the framework then compares real and synthetic distributions of node dependencies and attributes using Wasserstein-2 or total-variation distances. The CFG makes structure a checkable formal property: it lets the benchmark measure 'did the output follow the shape of the data' separately from 'is the content semantically good'.

What would settle it

Run a privacy-preserving generator at $\epsilon=4$ on the ICLR dataset and check whether it can simultaneously reach a CFG pass rate above $0.95$ and a KNN-recall above $0.8$ under the paper's grammar and embedding setup; a single such result would refute the claim that current DP generators cannot preserve structure and semantics together.

Watch

Extended reading notes

Core claim

The central discovery reported here is two-sided. First, no single evaluation metric fully describes the quality of DP synthetic data for structured, natural-language datasets: a sample that is semantically fluent can be structurally invalid, and a sample that satisfies a grammar can be semantically hollow. The paper demonstrates this by separating structural metrics from content metrics, and shows, for example, that instruction-following and Private Evolution reach high CFG pass rates on tabular and synthetic datasets while their semantic diversity (KNN-recall) stays near zero. Second, none of the state-of-the-art DP generators evaluated—Private Evolution, instruction following, and DP fine-tuning—reliably captures structure without sacrificing semantic performance; DP fine-tuning in particular achieves a CFG pass rate of zero on all datasets at $\epsilon=4$. The authors state this as a motivation for multi-metric benchmarking and for further research on structure-aware privacy-preserving generation.

Load-bearing premise

The benchmark's conclusions rest on the assumption that a user-provided context-free grammar captures the structural properties that matter for downstream use, and that embedding cosine similarity faithfully measures semantic dependency; if either fails, the rankings the benchmark produces could mislead.

Editorial extensions

If this is right

  • If synthetic text is meant for downstream pipelines, structural validity has to be a first-class evaluation axis, because a high single-similarity score can mask output that no parser or downstream utility would accept.
  • DP methods that rely on fine-tuning small open models are unlikely, on their own, to produce schema-compliant structured text; instruction-guided generation or training-free methods such as Private Evolution are needed to reach usable pass rates.
  • The benchmark's metric profile can be read diagnostically: low CFG pass rate points to formatting or schema loss, low KNN-recall points to diversity loss, and high key-node-dependency distance points to broken semantic correlations.
  • The described modifications to Private Evolution—reformatting invalid samples after private voting and extracting one node type while auto-generating the rest—are direct recipes for improving both structural validity and semantic diversity without extra privacy cost under the post-processing property.
  • A common multi-metric protocol would make DP structured-text generators comparable on the same yardstick, potentially shifting research incentives from single-score optimization toward balanced improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper stops short of claiming that CFG-based structure is sufficient for all structured text; a natural extension is to test whether context-sensitive constraints that a CFG cannot express, such as long-range agreement between non-adjacent nodes, produce failures that the benchmark's current metrics under-report.
  • Because the benchmark treats the user-provided grammar as a given, a useful next experiment would be to vary the grammar itself: if two equally reasonable grammars for the same dataset rank the same generators differently, then benchmark conclusions are grammar-sensitive and should be reported together with the grammar.
  • The metric suite could be reused outside differential privacy, for example to evaluate synthetic training data for data augmentation or conditional generation, where structural fidelity and semantic diversity are also the properties that matter; the paper mentions this possibility only in passing.
  • One could add a privacy side to the benchmark, for example measuring how much verbatim memorization of private samples survives in high-scoring synthetic outputs, to check that high quality scores do not come from copying; the current metrics do not assess this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes Struct-Bench, an evaluation framework and benchmark for differentially private (DP) synthetic data generation on structured datasets that contain natural language fields. The framework requires a user-supplied Context-Free Grammar (CFG) to parse each sample into nodes, and it defines three classes of metrics: structural (CFG Pass Rate, Key Node Dependency, Attribute Match), non-structural (KNN-Precision/Recall), and downstream task accuracy (TSTR). The benchmark includes seven datasets (ShareGPT, ICLR 2024 reviews, Water, Arena, Adult, Synthetic Reviews, Synthetic Grounding), each annotated with a CFG and key-node pairs. The authors benchmark several DP generators (Private Evolution, Instruction Following, DP fine-tuning, and non-private fine-tuning variants) under various privacy budgets, and report that no single metric fully describes synthetic data quality and that existing SOTA methods fail to capture structure without sacrificing semantic performance. A case study then uses Struct-Bench to improve Private Evolution via LLM-assisted CFG reformatting, node extraction and auto-generation, and format-token fixing, reporting improved CFG-PR, KNN-Precision, KNN-Recall, and several attribute-match metrics.

Significance. If the technical concerns are resolved, Struct-Bench would be a valuable community resource: it is, to my knowledge, the first benchmark focused specifically on DP synthetic data for structured datasets with natural-language components. The paper ships a public implementation and leaderboard, provides CFG annotations for seven diverse datasets, and makes a credible case that structure and content quality are orthogonal and need to be measured together. The multi-metric analysis is a useful correction to single-number evaluations. The case study also demonstrates a concrete workflow for using the benchmark to drive algorithm development. The main threats to this significance are the unvalidated and partly duplicate Key Node Dependency metric, the asymmetric foundation-model choices in the headline method comparison, and the incorrect or incomplete privacy accounting for DP-SGD with δ=0.

major comments (6)
  1. [Section 3.2, privacy accounting] The paper states in Section 3.2 that all experiments set δ=0. Since DP-FT is implemented with DP-SGD, which relies on Gaussian noise, an (ε,0)-DP guarantee is not achievable with the standard mechanism; either the privacy accounting is incorrect or the statement omits the actual δ value (e.g., the moments accountant result). Please specify the exact mechanism, the privacy accountant, and the realized δ per dataset, or rerun the DP-FT experiments with correctly accounting for δ > 0. This is necessary for the benchmark's privacy claims to be valid.
  2. [Section 2.1, Eq. (KND) / Table 5] KND is defined as the Wasserstein-2 distance between distributions of cosine similarities of embeddings of a key node pair (Oi, Oj). Cosine similarity is symmetric, so the distributions for (query, response) and (response, query) pairs are identical; the benchmark nonetheless reports them as separate metrics in Table 5 and in Figures 9, 15, 17, 20, and 21. Please either replace the scalar cosine similarity with an asymmetric dependency function (e.g., directional conditional likelihood) or remove the duplicated pairs, since the current metric suite overstates the number of independent signals.
  3. [Section 2.1 / Appendix A.1] The headline finding that DP methods cannot preserve structure without sacrificing semantics rests on KND as a measure of semantic dependency. On the Synthetic Grounding dataset, the key node pair (source1, source2) is designed to vary in logical consistency, but cosine similarity captures topical similarity rather than logical consistency, so contradictory sources can have cosine similarity distributions close to consistent ones. The same issue arises for (text, sentiment) in Reviews, where positive and negative reviews are topically close. Section 2 explicitly notes that a context-sensitive grammar would be needed for semantic dependencies, yet no validation (e.g., correlation of KND with human or LLM judgments on any of the seven datasets) is provided. Please add a validation study or sharply qualify the semantic-structure conclusions.
  4. [Table 6 and Section 3.3] In Table 6, PE and IF are run with GPT-4o while FT and DP-FT are limited to GPT-2, confounding method with foundation model. The conclusion that 'none of the existing SOTA DP methods are able to reliably capture the structural properties of data without sacrificing semantic performance' is therefore not directly supported as a statement about DP techniques. The authors already acknowledge this confound in the last bullet of Section 3.3, but the abstract and conclusion should be rephrased, and ideally same-model comparisons (e.g., PE with GPT-2 for all datasets, or DP-FT with a larger open-weights model) should be reported.
  5. [Tables 2 and 6] No variance estimates are reported for the main benchmark results. Given the stochasticity of DP noise, LLM sampling, and SGD fine-tuning, point estimates without error bars cannot support small quantitative comparisons (e.g., CFG-PR 0.17 vs 0.19 on ICLR, KND differences of 0.02). Please report means and standard deviations over at least three independent runs for the central tables, or explicitly state that the numbers are from a single run and soften the corresponding comparative claims.
  6. [Section 2.1 and Appendix A] The benchmark metrics are not fully reproducible from the manuscript because key implementation details are missing: the embedding model and k value for KNN-Precision/KNN-Recall, the embedding model for KND, the number of samples used to estimate the Wasserstein-2 distances, and the LLM/prompts used for AM semantic attributes such as topic and intent are not specified. Please add a complete specification in the appendix, even if the code is public.
minor comments (5)
  1. [Section 4, Problem 2] The sentence 'KNN-Precision of vanilla PE increases from 0.56 to 0.69' appears twice in succession; please remove the duplicate.
  2. [Section D.4, Figure 16] The label 'KNS' in Figure 16 should presumably be 'KND'.
  3. [Figure 2] The dataset names in Figure 2 are garbled (e.g., '6KDUH*37'), seemingly because of a font encoding problem; please regenerate the figure with standard text rendering.
  4. [Section C.3] The heading refers to Llama2-7b; please verify the model name/version and use consistent spelling.
  5. [Section B.4 / Appendix A] Please specify whether the semantic attributes used in AM (topic, intent, sentiment) are obtained from GPT-4o or another labeler, and include the exact labeling prompt, since these are part of the benchmark protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark metrics are explicit, empirical findings are read from tables, and case-study interventions transparently target the metrics they report.

full rationale

Struct-Bench is an evaluation framework rather than a derivational claim. The metrics (CFG-PR, KND, AM, KNN-Precision/Recall, TSTR) are defined explicitly in Sec. 2.1 and App. A and are computed from real and synthetic datasets; the paper's findings in Sec. 3.3 are read off Tables 2/6 and Fig. 2, with no fitted parameter renamed as a prediction and no definitional identity between an input and a claimed output. The PE case study in Sec. 4 does optimize benchmark metrics (e.g., 'we introduce a reformatting feature to the Random and Variation APIs by prompting LLMs to explicitly check and reformat CFG-invalid samples', which by construction raises CFG-PR), but the paper presents this transparently as a generator intervention and simultaneously reports non-targeted semantic/statistic/downstream metrics that could have contradicted the improvement. Self-citations to prior Private Evolution papers are baseline references only; no load-bearing uniqueness theorem or self-citation chain forces the conclusions. Concerns that KND's cosine-similarity proxy may miss logical consistency are about metric validity, not circularity. Hence no circular step is exhibited and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the adequacy of the CFG representation, the validity of embedding-based semantic proxies, and the reliability of LLM-generated labels. These are domain assumptions, not derivations. The case study introduces hand-picked hyperparameters such as the 65% fixed-token ratio.

free parameters (3)
  • k for KNN-Precision/Recall = Unspecified
    The neighbor count k is used in Section 2.1 but its value is not stated; results depend on it.
  • Fixed Selected Token ratio = 0.65
    In Appendix D.4 the 65% ratio is chosen by hand; no sensitivity analysis is reported.
  • Embedding model for KND and KNN metrics = Unspecified
    The embedding model used for cosine similarity is not identified in the paper, so the metric values are not uniquely reproducible.
assumptions (5)
  • domain assumption A user-provided Context-Free Grammar suffices to represent the structural constraints of a dataset
    The whole evaluation protocol depends on parsing samples with user-specified CFGs, introduced in Section 2.
  • domain assumption Cosine similarity between node embeddings captures semantic dependency between key node pairs
    KND in Section 2.1 uses embedding cosine similarity as the dependency function.
  • domain assumption LLM-generated labels from GPT-4o are reliable enough for downstream evaluation
    Label generation in Section 2.1 and Appendix A.3 relies on GPT-4o to produce ground-truth labels for TSTR accuracy.
  • domain assumption Train-synthetic-test-real (TSTR) accuracy is a valid measure of downstream utility
    Downstream evaluation in Section 2.1 uses TSTR without validating against train-real-test-real on all datasets.
  • standard math Post-processing property of differential privacy applies to the reformatting and node-extraction steps in Section 4
    This is a standard theorem of DP, invoked in Section 4 to claim no additional privacy cost.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Struct-Bench: A Benchmark for Differentially Private Structured Text Generation." pith.science (2026). https://pith.science/paper/FZLILACL

@misc{pith2026250910696,
  author       = {Pith},
  title        = {Pith review of: Struct-Bench: A Benchmark for Differentially Private Structured Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZLILACL}},
  note         = {Machine review of arXiv:2509.10696}
}
read the original abstract

Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at https://struct-bench.github.io.

Figures

Figures reproduced from arXiv: 2509.10696 by the authors.

Figure 1
Figure 1. Dataset level and sample level views into the Struct-Bench framework. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CFG-PR and KNN-Recall of baselines on Struct-Bench with different datasets. While frontier [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. CFG-PR of vanilla PE on ShareGPT. Each data point is averaged over three independent trials. CFG-PR is low for all ϵ. We mainly focus on improving structural validity and semantic diversity of the PE synthetic data in this section, and defer a more thorough analysis as well as methods on improving node dependency (KND) to §D. Problem 1: Structural Validity (CFG-PR) is low. Struc￾tural validity, i.e., CFG-PR, is a cr… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: CFG-PR of vanilla PE and PE with CFG reformat under ϵ = 4. As described in §3.2, PE generates a set of samples and then select those with high qualities by private voting in each iteration. Sample reformatting can happen before or after the PE private voting process. W…
Figure 5
Figure 5. Figure 5: Illustration of reformatting-before-voting on the ShareGPT dataset. The syntactically-correctly [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: KNN-Precision and KNN-Recall of vanilla PE on ShareGPT under different privacy guarantees. Both are low. increases from 0.56 to 0.69, while KNN-Recall remains very low, around 0.35. This suggests that vanilla PE focuses on semantic quality while sacrificing diversity. …
Figure 7
Figure 7. Figure 7: Example executions of the Variation API of vanilla PE and PE with node extraction and auto [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Performance of vanilla PE and PE with node extraction on KNN-Precision & KNN-Recall and [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 10
Figure 10. Figure 10: Illustration of the data modeling of ShareGPT. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Performance of PE and DP-FT on all metrics under different privacy budgets on ShareGPT. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Performance of PE and DP-FT on all metrics under different privacy budgets on ICLR. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Performance of different baselines on ShareGPT using Llama2-7b with [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Performance of Vanilla PE with different privacy guarantees under ShareGPT dataset [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Performance of PE with CFG Reformat on ShareGPT with [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Performance of Vanilla PE and PE with fix token on CFG-PR and KND [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Performance of PE with Fix Format Token on ShareGPT with [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Performance of vanilla PE and PE with node extraction on Type to Token Ratio (TTR). [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Embedding distributions of Vanilla PE and PE with node extraction. [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Performance of PE with Node Extraction on ShareGPT with [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: Performance of Different Methods on ShareGPT with [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 49 canonical work pages

  1. [1]

    Hugging Face Datasets https:// huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/ bcd32a724d8460ebe14e1d05b0195e30e9a46cb1, apr 2023

    ShareGPT_Vicuna_unfiltered Dataset. Hugging Face Datasets https:// huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/ bcd32a724d8460ebe14e1d05b0195e30e9a46cb1, apr 2023

    2023

  2. [2]

    OpenReview.net, 2024

    The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  3. [3]

    Abadi, A

    M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016

    2016

  4. [4]

    Dp-2stage: Adapting language models as differentially private tabular data generators.arXiv preprint arXiv:2412.02467, 2024

    Tejumade Afonja, Hui-Po Wang, Raouf Kerkouche, and Mario Fritz. Dp-2stage: Adapting language models as differentially private tabular data generators.arXiv preprint arXiv:2412.02467, 2024

    arXiv 2024

  5. [5]

    Really useful synthetic data–a framework to evaluate the quality of differentially private synthetic data.arXiv preprint arXiv:2004.07740, 2020

    Christian Arnold and Marcel Neunhoeffer. Really useful synthetic data–a framework to evaluate the quality of differentially private synthetic data.arXiv preprint arXiv:2004.07740, 2020

    arXiv 2004

  6. [6]

    Adulthttps://doi.org/10.24432/C5XW20

    Barry Becker and Ronny Kohavi. Adulthttps://doi.org/10.24432/C5XW20. UCI Machine Learning Repository, 1996

    doi:10.24432/c5xw20 1996

  7. [7]

    Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  8. [8]

    A universal metric for robust evaluation of synthetic tabular data.IEEE Transactions on Artificial Intelligence, 5(1):300–309, 2022

    Vikram S Chundawat, Ayush K Tarun, Murari Mandal, Mukund Lahoti, and Pratik Narang. A universal metric for robust evaluation of synthetic tabular data.IEEE Transactions on Artificial Intelligence, 5(1):300–309, 2022

    2022

Show all 94 references
  1. [9]

    Conditional synthetic data generation for robust machine learning applications with limited pandemic data

    Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni- Vincentelli, and Costas J Spanos. Conditional synthetic data generation for robust machine learning applications with limited pandemic data. InProceedings of the AAAI Conference on Ar...

    2022

  2. [10]

    Effective data generation for imbalanced learning using conditional generative adversarial networks.Expert Systems with applications, 91:464–471, 2018

    Georgios Douzas and Fernando Bacao. Effective data generation for imbalanced learning using conditional generative adversarial networks.Expert Systems with applications, 91:464–471, 2018

    2018

  3. [11]

    Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning.arXiv preprint arXiv:2008.09202, 2020

    Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning.arXiv preprint arXiv:2008.09202, 2020

    2008 arXiv

  4. [12]

    The GEM benchmark: Natural language generation, its evaluation and metrics

    Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Crist...

    2021

  5. [13]

    Differentially private diffusion models generate useful synthetic images.arXiv preprint arXiv:2302.13861, 2023

    Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images.arXiv preprint arXiv:2302.13861, 2023

    2023 arXiv

  6. [14]

    A unified framework for quantifying privacy risk in synthetic data.arXiv preprint arXiv:2211.10459, 2022

    Matteo Giomi, Franziska Boenisch, Christoph Wehmeyer, and Borbála Tasnádi. A unified framework for quantifying privacy risk in synthetic data.arXiv preprint arXiv:2211.10459, 2022

    2022 arXiv

  7. [15]

    Benchmarking fraud detectors on private graph data.KDD, 2025

    Alexander Goldberg, Giulia Fanti, Nihar Shah, and Steven Wu. Benchmarking fraud detectors on private graph data.KDD, 2025

    2025

  8. [16]

    Dpimagebench: A unified benchmark for differentially private image synthesis.arXiv preprint arXiv:2503.14681, 2025

    Chen Gong, Kecen Li, Zinan Lin, and Tianhao Wang. Dpimagebench: A unified benchmark for differentially private image synthesis.arXiv preprint arXiv:2503.14681, 2025

    2025 arXiv

  9. [17]

    An llm-based framework for synthetic data generation

    Mandeep Goyal and Qusay H Mahmoud. An llm-based framework for synthetic data generation. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC), pages 00340–00346. IEEE, 2025

    2025

  10. [18]

    Synthfair: Ensuring subgroup fairness in classification via synthetic data generation

    Begüm Hattatoğlu, Abdulhakim A Qahtan, Heysem Kaya, and Yannis Velegrakis. Synthfair: Ensuring subgroup fairness in classification via synthetic data generation. InWorld Congress in Computer Science, Computer Engineering & Applied Computing, pages 347–363. Springer, 2024

    2024

  11. [19]

    Synthetic tabular data evaluation in the health domain covering resemblance, utility, and privacy dimensions.Methods of information in medicine, 62(S 01):e19–e38, 2023

    Mikel Hernadez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic tabular data evaluation in the health domain covering resemblance, utility, and privacy dimensions.Methods of information in medicine, 62(S 01):e19–e38, 2023

    2023

  12. [20]

    Heusel, H

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. InAdvances in neural information processing systems, volume 30, 2017

    2017

  13. [21]

    Introduction to automata theory, languages, and computation.Acm Sigact News, 32(1):60–65, 2001

    John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. Introduction to automata theory, languages, and computation.Acm Sigact News, 32(1):60–65, 2001

    2001

  14. [22]

    Pre-text: training language models on private federated data in the age of llms

    Charlie Hou, Akshat Shrivastava, Hongyuan Zhan, Rylan Conway, Trang Le, Adithya Sagar, Giulia Fanti, and Daniel Lazar. Pre-text: training language models on private federated data in the age of llms. InProceedings of the 41st International Conference on Machine Learning, pages...

    2024

  15. [23]

    Private federated learning using preference-optimized synthetic data.arXiv preprint arXiv:2504.16438, 2025

    Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, and Giulia Fanti. Private federated learning using preference-optimized synthetic data.arXiv preprint arXiv:2504.16438, 2025

    2025 arXiv

  16. [24]

    Sok: Privacy-preserving data synthesis

    Yuzheng Hu, Fan Wu, Qinbin Li, Yunhui Long, Gonzalo Munilla Garrido, Chang Ge, Bolin Ding, David Forsyth, Bo Li, and Dawn Song. Sok: Privacy-preserving data synthesis. In2024 IEEE Symposium on Security and Privacy (SP), pages 4696–4713. IEEE, 2024

    2024

  17. [25]

    Kynkäänniemi, T

    T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  18. [26]

    Tregex and tsurgeon: Tools for querying and manipulating tree data structures

    Roger Levy and Galen Andrew. Tregex and tsurgeon: Tools for querying and manipulating tree data structures. InLREC, pages 2231–2234. Genoa, 2006

    2006

  19. [27]

    Z. Lin, S. Gopi, J. Kulkarni, H. Nori, and S. Yekhanin. Differentially private synthetic data via foundation model APIs 1: Images. InInternational Conference on Learning Representations (ICLR), 2024. 14

    2024

  20. [28]

    Differentially private synthetic data via apis 3: Using simulators instead of foundation model.arXiv preprint arXiv:2502.05505, 2025

    Zinan Lin, Tadas Baltrusaitis, Wenyu Wang, and Sergey Yekhanin. Differentially private synthetic data via apis 3: Using simulators instead of foundation model.arXiv preprint arXiv:2502.05505, 2025

    2025 arXiv

  21. [29]

    Differentially private synthetic data via foundation model apis 1: Images

    Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. Differentially private synthetic data via foundation model apis 1: Images. InICLR, 2024

    2024

  22. [30]

    Using gans for sharing networked time series data: Challenges, initial promise, and open questions

    Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar. Using gans for sharing networked time series data: Challenges, initial promise, and open questions. InProceedings of the ACM internet measurement conference, pages 464–483, 2020

    2020

  23. [31]

    Summary statistic privacy in data sharing

    Zinan Lin, Shuaiqi Wang, Vyas Sekar, and Giulia Fanti. Summary statistic privacy in data sharing. IEEE Journal on Selected Areas in Information Theory, 5:369–384, 2024

    2024

  24. [32]

    Preserving privacy in healthcare: A systematic review of deep learning approaches for synthetic data generation.Computer Methods and Programs in Biomedicine, page 108571, 2024

    Yintong Liu, U Rajendra Acharya, and Jen Hong Tan. Preserving privacy in healthcare: A systematic review of deep learning approaches for synthetic data generation.Computer Methods and Programs in Biomedicine, page 108571, 2024

    2024

  25. [33]

    An evaluation framework for synthetic data generation models

    Ioannis E Livieris, Nikos Alimpertis, George Domalis, and Dimitris Tsakalidis. An evaluation framework for synthetic data generation models. InIFIP International Conference on Artificial Intelligence Applications and Innovations, pages 320–335. Springer, 2024

    2024

  26. [34]

    Evaluating inter-column logical relationships in synthetic tabular data generation.arXiv preprint arXiv:2502.04055, 2025

    Yunbo Long, Liming Xu, and Alexandra Brintrup. Evaluating inter-column logical relationships in synthetic tabular data generation.arXiv preprint arXiv:2502.04055, 2025

    2025 arXiv

  27. [35]

    PhD thesis, Politecnico di Torino, 2025

    Maria Antonietta Longo.A Synthetic Data Generation Approach for Subgroup-Based Bias Mitigation in Structured Data. PhD thesis, Politecnico di Torino, 2025

    2025

  28. [36]

    Aim: An adaptive and iterative mechanism for differentially private synthetic data.arXiv preprint arXiv:2201.12677, 2022

    Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. Aim: An adaptive and iterative mechanism for differentially private synthetic data.arXiv preprint arXiv:2201.12677, 2022

    2022 arXiv

  29. [37]

    Benchmarking evaluation protocols for classifiers trained on differentially private synthetic data.IEEE Access, 2024

    Parisa Movahedi, Valtteri Nieminen, Ileana Montoya Perez, Hiba Daafane, Dishant Sukhwal, Tapio Pahikkala, and Antti Airola. Benchmarking evaluation protocols for classifiers trained on differentially private synthetic data.IEEE Access, 2024

    2024

  30. [38]

    Safesynthdp: Leveraging large language models for privacy-preserving synthetic data generation using differential privacy.arXiv preprint arXiv:2412.20641, 2024

    Md Mahadi Hasan Nahid and Sadid Bin Hasan. Safesynthdp: Leveraging large language models for privacy-preserving synthetic data generation using differential privacy.arXiv preprint arXiv:2412.20641, 2024

    2024 arXiv

  31. [39]

    Synthetic data for privacy-preserving clinical risk prediction.Scientific Reports, 14(1):25676, 2024

    Zhaozhi Qian, Thomas Callender, Bogdan Cebere, Sam M Janes, Neal Navani, and Mihaela van der Schaar. Synthetic data for privacy-preserving clinical risk prediction.Scientific Reports, 14(1):25676, 2024

    2024

  32. [40]

    Evaluating differentially private synthetic data generation in high-stakes domains.arXiv preprint arXiv:2410.08327, 2024

    Krithika Ramesh, Nupoor Gandhi, Pulkit Madaan, Lisa Bauer, Charith Peris, and Anjalie Field. Evaluating differentially private synthetic data generation in high-stakes domains.arXiv preprint arXiv:2410.08327, 2024

    2024 arXiv

  33. [41]

    Type/token ratios: What do they really tell us?Journal of child language, 14(2):201–209, 1987

    Brian Richards. Type/token ratios: What do they really tell us?Journal of child language, 14(2):201–209, 1987

    1987

  34. [42]

    Differ- entially private synthetic data: Applied evaluations and enhancements.arXiv preprint arXiv:2011.05537, 2020

    Lucas Rosenblatt, Xiaoyan Liu, Samira Pouyanfar, Eduardo de Leon, Anuj Desai, and Joshua Allen. Differ- entially private synthetic data: Applied evaluations and enhancements.arXiv preprint arXiv:2011.05537, 2020

    2011 arXiv

  35. [43]

    Assessing generative models via precision and recall.Advances in neural information processing systems, 31, 2018

    Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall.Advances in neural information processing systems, 31, 2018

    2018

  36. [44]

    Generating synthetic data with formal privacy guarantees: State of the art and the road ahead.arXiv preprint arXiv:2503.20846, 2025

    Viktor Schlegel, Anil A Bharath, Zilong Zhao, and Kevin Yee. Generating synthetic data with formal privacy guarantees: State of the art and the road ahead.arXiv preprint arXiv:2503.20846, 2025. 15

    2025 arXiv

  37. [45]

    Ai for data science: A benchmark for differentially private text dataset generators

    Viktor Schlegel, Yuping Wu, Warren Del-Pinto, Goran Nenadic, and Anil Anthony Bharath. Ai for data science: A benchmark for differentially private text dataset generators. InAI4X 2025 International Conference

    2025

  38. [46]

    On the foundations of quantitative information flow

    Geoffrey Smith. On the foundations of quantitative information flow. InInternational Conference on Foundations of Software Science and Computational Structures, pages 288–302. Springer, 2009

    2009

  39. [47]

    Evaluation is key: a survey on evaluation measures for synthetic time series.Journal of Big Data, 11(1):66, 2024

    Michael Stenger, Robert Leppich, Ian Foster, Samuel Kounev, and André Bauer. Evaluation is key: a survey on evaluation measures for synthetic time series.Journal of Big Data, 11(1):66, 2024

    2024

  40. [48]

    Formalizing and estimating distribution inference risks.arXiv preprint arXiv:2109.06024, 2021

    Anshuman Suri and David Evans. Formalizing and estimating distribution inference risks.arXiv preprint arXiv:2109.06024, 2021

    2021 arXiv

  41. [49]

    Synthesizing privacy-preserving text data via finetuning without finetuning billion-scale llms.arXiv preprint arXiv:2503.12347, 2025

    Bowen Tan, Zheng Xu, Eric Xing, Zhiting Hu, and Shanshan Wu. Synthesizing privacy-preserving text data via finetuning without finetuning billion-scale llms.arXiv preprint arXiv:2503.12347, 2025

    2025 arXiv

  42. [50]

    Privacy-preserving in-context learning with differentially private few-shot generation.arXiv preprint arXiv:2309.11765, 2023

    Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially private few-shot generation.arXiv preprint arXiv:2309.11765, 2023

    2023 arXiv

  43. [51]

    Benchmarking differentially private synthetic data generation algorithms.arXiv preprint arXiv:2112.09238, 2021

    Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Benchmarking differentially private synthetic data generation algorithms.arXiv preprint arXiv:2112.09238, 2021

    2021 arXiv

  44. [52]

    Water Bottle Dataset - Flipkart https://www.kaggle.com/datasets/tharunmss/ water-bottle-dataset-flipkart

    Tharunmss. Water Bottle Dataset - Flipkart https://www.kaggle.com/datasets/tharunmss/ water-bottle-dataset-flipkart. Kaggle, 2024

    2024

  45. [53]

    Kajal: Extracting grammar of a source code using large language models

    Mohammad Jalili Torkamani. Kajal: Extracting grammar of a source code using large language models. arXiv preprint arXiv:2412.08842, 2024

    2024 arXiv

  46. [54]

    Dp-cgan: Differentially private synthetic data and label generation

    Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019

    2019

  47. [55]

    Differentially private tabular data synthesis using large language models

    Toan V Tran and Li Xiong. Differentially private tabular data synthesis using large language models. arXiv preprint arXiv:2406.01457, 2024

    2024 arXiv

  48. [56]

    Synthetic data, real errors: how (not) to publish and use synthetic data

    Boris Van Breugel, Zhaozhi Qian, and Mihaela Van Der Schaar. Synthetic data, real errors: how (not) to publish and use synthetic data. InInternational Conference on Machine Learning, pages 34793–34808. PMLR, 2023

    2023

  49. [57]

    Synthesize privacy-preserving high-resolution images via private textual intermediaries.arXiv preprint arXiv:2506.07555, 2025

    Haoxiang Wang, Zinan Lin, Da Yu, and Huishuai Zhang. Synthesize privacy-preserving high-resolution images via private textual intermediaries.arXiv preprint arXiv:2506.07555, 2025

    2025

  50. [58]

    Statistic maximal leakage

    Shuaiqi Wang, Zinan Lin, and Giulia Fanti. Statistic maximal leakage. In2024 IEEE International Symposium on Information Theory (ISIT), pages 2742–2747. IEEE, 2024

    2024

  51. [59]

    dp-transformers: Training transformer models with differential privacy, 2022

    Lukas Wutschitz, Huseyin A Inan, and Andre Manoel. dp-transformers: Training transformer models with differential privacy, 2022

    2022

  52. [60]

    Differentially private synthetic data via foundation model apis 2: Text

    Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, et al. Differentially private synthetic data via foundation model apis 2: Text. InInternational Conference on Machine Learning, pages 54531–54...

    2024

  53. [61]

    Generation and evaluation of privacy preserving synthetic health data.Neurocomputing, 416:244–255, 2020

    Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P Bennett. Generation and evaluation of privacy preserving synthetic health data.Neurocomputing, 416:244–255, 2020

    2020

  54. [62]

    Structured evaluation of synthetic tabular data.arXiv preprint arXiv:2403.10424, 2024

    Scott Cheng-Hsin Yang, Baxter Eaves, Michael Schmidt, Ken Swanson, and Patrick Shafto. Structured evaluation of synthetic tabular data.arXiv preprint arXiv:2403.10424, 2024. 16

    2024 arXiv

  55. [63]

    Differentially private fine-tuning of language models.arXiv preprint arXiv:2110.06500, 2021

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models.arXiv preprint arXiv:2110.06500, 2021

    arXiv

  56. [64]

    A multi-faceted evaluation framework for assessing synthetic data generated by large language models.arXiv preprint arXiv:2404.14445, 2024

    Yefeng Yuan, Yuhong Liu, and Liang Cheng. A multi-faceted evaluation framework for assessing synthetic data generated by large language models.arXiv preprint arXiv:2404.14445, 2024

    2024 arXiv

  57. [65]

    X. Yue, H. A. Inan, X. Li, G. Kumar, J. McAnallen, H. Sun, D. Levitan, and R. Sim. Synthetic text generation with differential privacy: A simple and practical recipe. InACL, 2023

    2023

  58. [66]

    In30th USENIX Security Symposium (USENIX Security 21), pages 929–946, 2021

    Zhikun Zhang, Tianhao Wang, Ninghui Li, Jean Honorio, Michael Backes, Shibo He, Jiming Chen, and Yang Zhang.{PrivSyn}: Differentially private data synthesis. In30th USENIX Security Symposium (USENIX Security 21), pages 929–946, 2021

    2021

  59. [67]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena, 2023

    2023

  60. [68]

    Which nodes are central to our downstream tasks, and which nodes are semantically related to them?

    Tianyuan Zou, Yang Liu, Peng Li, Yufei Xiong, Jianqing Zhang, Jingjing Liu, Xiaozhou Ye, Ye Ouyang, and Ya-Qin Zhang. Contrastive private data synthesis via weighted multi-plm fusion.arXiv preprint arXiv:2502.00245, 2025. 17 A Metric Definitions and Instantiation Guidelines A....

    2025 arXiv

  61. [70]

    (response, query) pair AM

  62. [71]

    number of nodes 2.query token length

  63. [72]

    response token length

  64. [73]

    intent prediction ICLR CFG-PR KND

  65. [74]

    (review, rebuttal) pair

  66. [75]

    (rebuttal, comment) pair

  67. [76]

    (review, review) pair from different reviewers AM

  68. [77]

    rebuttal token length

  69. [78]

    KNN-Recall topic prediction Arena CFG-PR KND

  70. [79]

    (conversation_a, conversation_b) pair AM

  71. [80]

    KNN-Recall winner prediction Water CFG-PR KND

  72. [81]

    (title, cleaned_review) pair AM

  73. [82]

    KNN-Recall rating prediction Adult CFG-PR KND

  74. [83]

    (native country, workclass) pair AM

  75. [84]

    KNN-Recall income prediction Reviews CFG-PR KND

  76. [85]

    (text, sentiment) pair AM

  77. [86]

    KNN-Recall review label prediction Grounding CFG-PR KND

  78. [87]

    (source1, source2) pair AM

  79. [88]

    We prepend the instructions to each training sample and fine-tune the foundation model for 20 epochs with batch size 32, weight decay 0.01, and learning rate10−4

    KNN-Recall query relevancy prediction 5 Implementation Details on Instruction Fine-tuningFor both Instruct DP-FT and Instruct FT, we use the same instructions as those in the Random API of PE. We prepend the instructions to each training sample and fine-tune the foundation mod...

    1900

  80. [89]

    number of statements

  81. [90]

    response token length KND:

  82. [91]

    (query, response) pair

  83. [92]

    (response, query) pair AM:

  84. [93]

    KNN-Precision

    intent CFG-PR Non-structural Metrics - 1. KNN-Precision

  85. [94]

    topic prediction

    KNN-Recall - Downstream Tasks - 1. topic prediction

  86. [95]

    HUMAN:␣", and ChatGPT response must start with

    intent prediction - We illustrate and compare the performance of PE with privacy parameterϵ∈{1,2,4,∞} under structural semantic and statistic metrics in Figs. 14c and 14d respectively, and plot the CFG-PR and KNN-Precision & KNN-Recall in Figs. 14a and 14b. We do not include P...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.