Pith. sign in

REVIEW 9 cited by

Toward Training at ImageNet Scale with Differential Privacy

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2201.12328 v2 pith:FNRMSRHD submitted 2022-01-28 cs.LG

Alexey Kurakin , Shuang Song , Steve Chien , Roxana Geambasu , Andreas Terzis , Abhradeep Thakurta This is my paper · ORCID
classification cs.LG
keywords privacytrainingaccuracydifferentialimagenetmodelsscaletrain
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Differential privacy (DP) is the de facto standard for training machine learning (ML) models, including neural networks, while ensuring the privacy of individual examples in the training set. Despite a rich literature on how to train ML models with differential privacy, it remains extremely challenging to train real-life, large neural networks with both reasonable accuracy and privacy. We set out to investigate how to do this, using ImageNet image classification as a poster example of an ML task that is very challenging to resolve accurately with DP right now. This paper shares initial lessons from our effort, in the hope that it will inspire and inform other researchers to explore DP training at scale. We show approaches that help make DP training faster, as well as model types and settings of the training process that tend to work better in the DP setting. Combined, the methods we discuss let us train a Resnet-18 with DP to $47.9\%$ accuracy and privacy parameters $\epsilon = 10, \delta = 10^{-6}$. This is a significant improvement over "naive" DP training of ImageNet models, but a far cry from the $75\%$ accuracy that can be obtained by the same network without privacy. The model we use was pretrained on the Places365 data set as a starting point. We share our code at https://github.com/google-research/dp-imagenet, calling for others to build upon this new baseline to further improve DP at scale.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Correlating Cross-Iteration Noise for DP-SGD using Model Curvature

    cs.LG 2025-10 conditional novelty 7.0 of 10

    Using Hessian eigenvalues from public data to design correlated noise for DP-SGD improves accuracy by 1–4% over current DP-MF methods.

  2. Boosting Gradient Leakage Attacks: Data Reconstruction in Realistic FL Settings

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FedLeak reconstructs high-fidelity images from federated learning gradients at practical batch sizes, without auxiliary data, by matching only the largest gradient components and regularizing the optimization.

  3. Scaling Laws for Differentially Private Language Models

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Differentially private language models obey scaling laws in which compute-optimal models are roughly 10-50x smaller than non-private Chinchilla-optimal models, with large batch sizes and rapid saturation of compute.

  4. Lower Bounds for Public-Private Learning under Distribution Shift

    cs.LG 2025-07 reject novelty 6.0 of 10

    For Gaussian mean estimation and linear regression with distribution shift, the paper claims that public data never provides complementary value: either public data alone suffices, or (for large shifts) private data a...

  5. Hyperparameters in Score-Based Membership Inference Attacks

    cs.LG 2025-02 accept novelty 6.0 of 10

    A new shadow-model hyperparameter selection method (KL-LiRA) makes membership inference attacks nearly as effective without knowing target hyperparameters, and training-data-based hyperparameter tuning shows no detect...

  6. On the Convergence of DP-SGD with Adaptive Clipping

    cs.LG 2024-12 conditional novelty 6.0 of 10

    SGD with quantile clipping has an inherent bias that a time-varying quantile and step size schedule can remove, and the same analysis extends to differentially private training.

  7. Protecting Confidentiality, Privacy and Integrity in Collaborative Learning

    cs.DC 2024-12 reject novelty 6.0 of 10

    Citadel++ claims to protect dataset, model, and code confidentiality, user-level differential privacy, and execution integrity in collaborative training using VM-level trusted execution environments and enhanced DP-SGD.

  8. The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis

    cs.CV 2026-01 reject novelty 5.0 of 10

    In DP-SGD chest X-ray classification, MIMIC-CXR supervised pretraining beats ImageNet and DINOv3 initializations, but the study cannot cleanly separate pretraining domain from objective because key comparison arms are...

  9. The Impact of Generalization Techniques on the Interplay Among Privacy, Utility, and Fairness in Image Classification

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Combining DP-SAT with De et al.'s generalization techniques reaches 81.11% accuracy under (8, 10^-5)-DP on CIFAR-10, while amplifying bias on skewed datasets.

Pith tools