REVIEW 18 cited by
Unlocking High-Accuracy Differentially Private Image Classification through Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Unlocking High-Accuracy Differentially Private Image Classification through Scale
read the original abstract
Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points. Differentially Private Stochastic Gradient Descent (DP-SGD), the most popular DP training method for deep learning, realizes this protection by injecting noise during training. However previous works have found that DP-SGD often leads to a significant degradation in performance on standard image classification benchmarks. Furthermore, some authors have postulated that DP-SGD inherently performs poorly on large models, since the norm of the noise required to preserve privacy is proportional to the model dimension. In contrast, we demonstrate that DP-SGD on over-parameterized models can perform significantly better than previously thought. Combining careful hyper-parameter tuning with simple techniques to ensure signal propagation and improve the convergence rate, we obtain a new SOTA without extra data on CIFAR-10 of 81.4% under (8, 10^{-5})-DP using a 40-layer Wide-ResNet, improving over the previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a remarkable 83.8% top-1 accuracy on ImageNet under (0.5, 8*10^{-7})-DP. Additionally, we also achieve 86.7% top-1 accuracy under (8, 8 \cdot 10^{-7})-DP, which is just 4.3% below the current non-private SOTA for this task. We believe our results are a significant step towards closing the accuracy gap between private and non-private image classification.
Forward citations
Cited by 18 Pith papers
-
ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?
ContinuousBench shows non-private synthetic text transfers corpus-specific capabilities while state-of-the-art DP methods fail to do so even at ε=100.
-
Private and Stable Test-Time Adaptation with Differential Privacy
Differential privacy versions of TTA methods achieve privacy on ImageNet-C with small accuracy cost and can improve stability via clipping in continual settings.
-
Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?
Balanced Iteration Subsampling achieves stronger privacy amplification than Poisson subsampling in DP-SGD by eliminating participation variance while keeping uniform marginal participation.
-
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
PACZero achieves zero mutual information privacy for LLM fine-tuning via sign-quantized zeroth-order gradients, delivering near-non-private accuracy on SST-2 and SQuAD at I=0.
-
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
PACZero achieves zero mutual information privacy in LLM fine-tuning via sign-quantized subset-aggregated ZO gradients, delivering near non-private accuracy on SST-2 at I=0.
-
FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction
FiBeR adds a closed-form filter-aware correction A(ω)σ_w² to the second-moment term for temporally filtered DP gradients, improving adaptive optimization performance.
-
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
DPrivBench shows that top LLMs handle basic differential privacy mechanisms but fail on advanced algorithms, exposing gaps in automated DP reasoning.
-
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
DPrivBench is a new benchmark for evaluating LLMs on differential privacy reasoning, with results showing good performance on textbook mechanisms but substantial failures on advanced algorithms.
-
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
DPQuant uses epoch-wise probabilistic layer rotation and DP loss sensitivity to quantize only a changing subset of layers, reducing accuracy degradation from quantization noise in DP-SGD and delivering up to 2.21x thr...
-
High-Dimensional Private Linear Regression with Optimal Rates
DP-GD achieves minimax optimal non-asymptotic risk O(γ + γ²/ρ²) for well-conditioned high-dimensional data and power-law scaling for ill-conditioned power-law spectra, with the exponent depending on the privacy parameter ρ.
-
Privacy Leakage via Output Label Space and Differentially Private Continual Learning
Identifies output label space as a privacy side-channel in DP continual learning, formalizes DP for CL, and demonstrates two mitigation methods yielding higher accuracy than prior work.
-
Differentially Private Natural Gradient Descent
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...
-
Private and Stable Test-Time Adaptation with Differential Privacy
Casting Tent, EATA, SAR, DeYO, and COME into DP-TTA via per-sample clipping and Gaussian noise yields adequate privacy on ImageNet-C at modest accuracy and compute cost, with clipping sometimes improving stability.
-
Cyclic Adaptive Private Synthesis for Sharing Real-World Data in Education
CAPS provides an iterative differentially private synthesis method that outperforms one-shot baselines on authentic educational real-world data.
-
Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGD
Shuffled DP-SGD requires σ ≥ 1/√(2 ln M) or κ ≥ (1/√8)(1 - 1/√(4π ln M)) to limit adversarial advantage, preventing strong privacy and high utility simultaneously.
-
Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy
Add/remove adjacency in DP overstates attribute privacy relative to substitute adjacency; new auditing attacks confirm inconsistency with add/remove reports but consistency with substitute accounting.
-
Memory-Efficient Differentially Private Training with Gradient Random Projection
DP-GRAPE reduces memory in differentially private neural network training by using random Gaussian projections on gradients instead of SVD, achieving comparable privacy-utility tradeoffs to DP-SGD and scaling to 6.7B ...
-
On Optimal Hyperparameters for Differentially Private Deep Transfer Learning
Empirical study of DP transfer learning reveals that larger clipping bounds outperform under tight privacy and cumulative DP noise explains batch-size effects better than existing heuristics.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.