REVIEW 6 cited by
Fine-Tuning Large Language Models with User-Level Differential Privacy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user. We study two variants of DP-SGD with: (1) example-level sampling (ELS) and per-example gradient clipping, and (2) user-level sampling (ULS) and per-user gradient clipping. We derive a novel user-level DP accountant that allows us to compute provably tight privacy guarantees for ELS. Using this, we show that while ELS can outperform ULS in specific settings, ULS generally yields better results when each user has a diverse collection of examples. We validate our findings through experiments in synthetic mean estimation and LLM fine-tuning tasks under fixed compute budgets. We find that ULS is significantly better in settings where either (1) strong privacy guarantees are required, or (2) the compute budget is large. Notably, our focus on LLM-compatible training algorithms allows us to scale to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.
Forward citations
Cited by 6 Pith papers
-
On the Inherent Privacy of Zeroth Order Projected Gradient Descent
Zeroth-order projected gradient descent without additive Gaussian noise is not differentially private in the worst case, and its privacy loss grows superlinearly with iterations.
-
Transforming Remanufacturing Automation with Large Language Models: A Forward-Looking Analysis with Case Studies
The authors propose ReManGPT, a conceptual orchestration framework for applying LLMs to remanufacturing, and illustrate it with case studies in disassembly planning, repair guidance, and robotic execution.
-
Enhancing Model Privacy in Federated Learning with Random Masking and Quantization
FedQSN hides part of the server model with random masks and quantizes the remainder to give clients a degraded proxy, reporting a large global-vs-proxy performance gap with modest loss in the final global model.
-
On Design Principles for Private Adaptive Optimizers
A theoretical and empirical study finds that unbiased second-moment estimates in private Adam can be harmful in high dimensions, and that scale-then-privatize outperforms the alternatives on a small transformer task.
-
Secure Multi-LLM Agentic AI and Agentification for Edge General Intelligence by Zero-Trust: A Survey
A survey proposing zero-trust architecture for multi-LLM systems in edge computing, with a taxonomy of model- and system-level defenses and a conceptual framework.
-
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
Discussion (0). Sign in to comment.