REVIEW 3 cited by
Scaling Laws for Differentially Private Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Scaling Laws for Differentially Private Language Models
read the original abstract
Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings.
Forward citations
Cited by 3 Pith papers
-
Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?
Balanced Iteration Subsampling achieves stronger privacy amplification than Poisson subsampling in DP-SGD by eliminating participation variance while keeping uniform marginal participation.
-
High-Dimensional Private Linear Regression with Optimal Rates
DP-GD achieves minimax optimal non-asymptotic risk O(γ + γ²/ρ²) for well-conditioned high-dimensional data and power-law scaling for ill-conditioned power-law spectra, with the exponent depending on the privacy parameter ρ.
-
Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD
Shuffled DP-SGD requires σ ≥ 1/√(2 ln M) or κ ≥ (1/√8)(1 - 1/√(4π ln M)) to limit adversarial advantage, preventing strong privacy and high utility simultaneously.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.