REVIEW 8 cited by
Harnessing large-language models to generate private synthetic text
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.
Forward citations
Cited by 8 Pith papers
-
Optimal Domain-Aware Privacy Mechanisms for Synthetic Data Generation
For histogram-based DP synthetic data, mixing the private histogram with a floor-raised version of a same-domain public distribution is asymptotically the best linear privacy mechanism.
-
Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data
A model-agnostic audit detects synthetic data disclosures via feature-match and membership-inference tests that separate true from phantom leaks and give empirical differential-privacy lower bounds.
-
Clustering and Median Aggregation Improve Differentially Private Inference
Clustering seed texts and privately aggregating median token logits improves representativeness and reduces reported privacy cost for DP synthetic text generation.
-
Language Agents as Digital Representatives in Collective Decision-Making
Fine-tuned language models can generate individual critiques that, when fed into a consensus-building process, yield outcomes close to those produced by real participants.
-
MAPLE: Metadata Augmented Private Language Evolution
MAPLE boosts Private Evolution's initial prompt with DP tabular metadata and donated in-context examples, improving synthetic-text utility, convergence, and API cost on specialized corpora.
-
Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applications
A production mobile-keyboard error-correction system that synthesizes large LLM-generated training data, reweights it with a differentially private federated small LM, and fine-tunes a billion-parameter LLM via LoRA r...
-
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
-
How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy
A practical, extremely thorough survey of differentially private synthetic data generation: methods, privacy units, evaluation metrics, and end-to-end system components across four data modalities.
Discussion (0). Continue with ORCID to comment.