REVIEW 2 cited by
Differentially Private Language Models Benefit from Public Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.
Forward citations
Cited by 2 Pith papers
-
FlashDP: Private Training Large Language Models with Efficient DP-SGD
FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.
-
Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
A framework for DP fine-tuning of MLLMs that prunes visual tokens before training and selectively applies noisy gradient updates to blocks with the largest norms, reporting modest utility and memory gains over DP-SGD.
Discussion (0). Sign in to comment.