Pith. sign in

REVIEW 2 major objections 3 minor 96 references

Private Direct Preference Optimization for LLM Alignment

T0 review · 2 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The central claim is that DPO can be made preference-private at production scale by perturbing only a one-dimensional preference axis, via an unbiased randomized rescaling of the objective.

desk verdict Genuinely clever mechanism with a sound formal core, but the public-text threat model is narrower than the motivating pitch and the experiments need error bars. read the letter →

arxiv 2608.05040 v1 pith:7WH4LKBM submitted 2026-08-05 cs.CR

Yangfan Jiang , Fei Wei , Ergute Bao , Xiaokui Xiao , Yaliang Li , Bolin Ding This is my paper
classification cs.CR
keywords directpreferenceoptimizationprivacylabeldifferentialLLMalignmentrandomizedrescalingaxisunbiasedperturbationprivacy-utilitytrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct preference optimization (DPO) aligns LLMs with human preferences, but the preference label itself can reveal an annotator's sensitive judgment. This paper argues that in standard DPO data the privacy unit that usually matters is only the preference ordering, since prompts and candidate responses are typically public or model-generated. It formalizes preference privacy, a label-DP-style guarantee that protects only the preference signal, and proposes PrivDPO, which adds calibrated randomness along a one-dimensional preference axis instead of to the full gradient. Because the perturbation is an unbiased rescaling of the DPO objective, training stays close to standard DPO while satisfying pure $\epsilon$-preference privacy. Experiments on three alignment benchmarks and LLMs up to 32B parameters show that the method tracks non-private DPO utility far better than DP-SGD or randomized-response baselines.

What carries the argument

The load-bearing object is the preference axis, the one-dimensional subspace spanned by $\boldsymbol{v}=\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)$, along which a preference flip moves the DPO gradient. The preference intensity $\psi=\sigma\bigl(\beta\log[\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)]-\beta\log[\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x)]\bigr)$ is the private scalar; flipping the preference shifts it from $\psi$ to $\psi-1$. PrivDPO perturbs $\psi$ with a two-point asymmetric mechanism that outputs $\psi+1/(e^\epsilon-1)$ with probability $e^\epsilon/(e^\epsilon+1)$ and $\psi-1-1/(e^\epsilon-1)$ otherwise, which keeps the expectation at $\psi$ while bounding the likelihood ratio by $e^\epsilon$. Because this perturbation is equivalent to multiplying the DPO objective by a stop-gradient random weight $\tilde{w}=\tilde{\psi}/\psi$, the mechanism reduces to one scalar rescaling in the forward pass, avoiding per-example gradient clipping or noise injection.

What would settle it

Compute the gradients of the DPO loss for a triplet and for its preference-reversed twin on a fixed checkpoint, and measure the norm of the difference after projecting out the preference axis $\boldsymbol{v}$; if that residual exceeds machine precision, the one-dimensional-axis claim fails and the privacy proof leaves components unprotected.

Watch

Extended reading notes

Core claim

The central claim is that the privacy leakage surface of DPO is one-dimensional: for two preference-neighboring triplets $(x,y_w,y_l)$ and $(x,y_l,y_w)$, the gradients differ only along the vector $\boldsymbol{v}=\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)$, which is fixed by the public text. All preference information passes through the scalar coefficient on this axis, so randomizing that scalar is sufficient to make the preference signal indistinguishable, and any component orthogonal to $\boldsymbol{v}$ carries no preference information. PrivDPO realizes this by an unbiased asymmetric rescaling of the DPO objective, and Theorem 4.1 proves that the released gradients satisfy pure $\epsilon$-preference privacy while Lemma 4.1 proves the gradient estimate is unbiased. The paper further claims this design is practical at scale because the rescaling is a forward-pass scalar operation and needs no per-example gradient.

Load-bearing premise

The guarantee only covers the preference ordering; it assumes the prompt and both candidate responses are non-sensitive and already known to the adversary, so if the text itself is private the method does not protect it.

Editorial extensions

If this is right

  • A dataset whose only sensitive component is the preference ordering can be aligned with a pure $\epsilon$-preference privacy guarantee, with no dependence on the parameter dimension $d$ in the privacy analysis.
  • The private update is unbiased, so in expectation PrivDPO follows the same trajectory as standard DPO; Theorem 4.2 gives high-probability concentration of the batch gradient around the non-private gradient.
  • Under the single-epoch protocol each example is used once, so the full procedure satisfies dataset-level $\epsilon$-preference privacy without within-epoch composition; multiple epochs compose linearly.
  • The private SFT stage trains symmetrically on both responses, so the complete pipeline has no preference privacy cost in SFT and a clean $\epsilon$ budget in DPO.
  • Empirically, PrivDPO achieves reward margins two to three times larger than randomized-response and scalar-perturbation baselines, and win rates close to non-private DPO, across three benchmarks and model sizes up to 32B parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same one-dimensional-axis test applies to other preference objectives: measuring whether gradient differences of IPO, SimPO, or KTO are collinear with $\boldsymbol{v}$ would show which of those losses admit a PrivDPO-style mechanism.
  • Beyond the paper, the guarantee is best understood as one component of a layered defense: for prompts or responses that are themselves sensitive, preference privacy would need to be composed with text-level differential privacy rather than used alone.
  • Beyond the paper, the empirical memorization attack is only a lower-bound sanity check; a stronger test would train a small model and measure an optimal adversary's posterior odds against the $e^\epsilon$ bound to see whether the mechanism is loose in practice.
  • Beyond the paper, the transformation-invariance proof requires the mechanism's random bits to stay secret, so production deployments should ensure that random seeds and sampling states are not exposed in checkpoints or logs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper formalizes ε-preference privacy for DPO, a label-DP-style notion that protects only the binary preference ordering between two candidate responses, under the assumption that the prompt and both responses are public or non-sensitive. It then proposes PrivDPO, which privatizes the DPO objective by rescaling each example's loss with a carefully calibrated random weight that is unbiased in expectation, avoiding per-example gradient clipping or full-gradient noise. The paper claims dataset-level ε-preference privacy for the single-epoch training protocol of Algorithm 1, proves unbiasedness and a concentration bound, and supports the method with experiments on Anthropic-HH, TL;DR, and UltraFeedback across Llama, Qwen, and Pythia families up to 32B parameters.

Significance. If the guarantees hold, this is a substantive and timely contribution. The per-example privacy mechanism is elegant: it exploits the fact that a preference flip changes the DPO gradient only along a one-dimensional preference axis, and the two-point randomized rescaling gives an exact e^ε likelihood ratio. The unbiasedness lemma follows cleanly from E[w]=1, and the mechanism is compatible with large-scale training because it avoids per-example gradients. The empirical evaluation is broad, spans three model families and up to 32B parameters, and the code is publicly released, which strengthens reproducibility. The low-rank spectral measurement is appropriately framed as supporting evidence rather than as a fitted input to the mechanism. The main caveats are the scope of the threat model and the formality of the dataset-level guarantee, both of which are addressable in revision.

major comments (2)
  1. [Section 1.1, Section 3.2] The public-text assumption is load-bearing for the entire framework, yet the motivating vertical domains named in Section 1.1 (healthcare, finance, legal assistance) are precisely settings where prompts and responses are often sensitive. The only defense offered is the statement in Section 3.2 that the assumption 'holds in many practical DPO settings,' and the three evaluation benchmarks are public, curated datasets that do not test the premise in those motivating domains. I am not asking for new experiments on proprietary data, but the paper should either provide concrete evidence or a principled argument that DPO pipelines in these verticals satisfy the public-text premise, or it should explicitly retract the suggestion that PrivDPO addresses privacy in sensitive-prompt settings. As written, the motivation could lead practitioners in the named domains to overestimate the protection actually provided.
  2. [Section 4.4, Theorem 4.1 and Remark] Theorem 4.1 is stated for Algorithm 1, but the proof in Appendix B.1 establishes ε-preference privacy only for the per-example mechanism M defined on lines 4-8. The dataset-level claim is relegated to a paragraph-length Remark and to the informal discussion in Section 3.1. To make the central claim rigorous, please state and prove a dataset-level theorem covering the full transcript: the random permutation, mini-batch averaging, the adaptive dependence of later gradients on earlier private releases, and the post-processing argument that releases after the affected batch are functions of the ε-private release plus unchanged public examples. The argument is plausible and can likely be made formal, but as it stands the statement 'Algorithm 1 satisfies ε-preference privacy' is stronger than what is proved.
minor comments (3)
  1. [Section 4.4, Eq. (7)] The DP-SGD error bound appears to have an incorrect normalization. For a batch of size m, averaging m Gaussian noise vectors gives N(0, σ²/m I_d), so ||g̃ - g*||₂ = O(σ√(d/m)) = O(√(d log(1/δ))/(ε√m)), not O(√(d log(1/δ))/(εm)). Please correct; the qualitative comparison with PrivDPO is not overturned because λ should be compared with the corrected dimension-dependent term.
  2. [Section 5.3 and Appendix C] The text says Duchi and Piecewise are compared 'under the same privacy budget,' but Appendix C explains that these mechanisms enforce a stronger privacy notion than ε-preference privacy. The comparison should be qualified in the main text so that the reported utility gaps are not read as a like-for-like comparison at the same guarantee.
  3. [Algorithm 1, lines 4-6] The mechanism divides by ψ, which can be arbitrarily close to zero. In finite-precision arithmetic the sigmoid can underflow to 0, so a numerical guard or a lower bound on ψ should be described in the implementation details to avoid division-by-zero or extreme weight inflation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PrivDPO's privacy and unbiasedness proofs are parameter-free and self-contained; the preference-axis reduction is a direct algebraic identity.

full rationale

The paper's derivation chain is self-contained. The mechanism's randomness is the two-point distribution in Algorithm 1 (line 6), whose offset 1/(e^eps-1) is chosen to force E[w̃|ψ]=1; this is a stated mathematical condition, not a fitted parameter. Theorem 4.1 is proven in Appendix B.1 by a direct case analysis of the likelihood ratio between preference-neighboring gradients (ratios e^eps and e^{-eps} on the two support points, zero outside), and Lemma 4.1 follows immediately from E[w̃|ψ]=1. The load-bearing geometric claim (Section 4.2) is an algebraic identity: substituting the sigmoid identity σ(z)=1-σ(-z) gives Λ=-βv, so the gradient difference is exactly one-dimensional. No step invokes a self-citation to justify the central privacy or unbiasedness results; references to the authors' prior work ([8], [42]) are contextual related-work citations. The low-rank spectral observation in Section 4.4 is explicitly labeled as representative empirical support rather than an input to the mechanism, so it is not a fitted-input-called-prediction. The public-prompt/response assumption in Section 3.2 is a threat-model scope limitation, not a circular reduction of the theorem to its premise. I find no step where a claimed output is equivalent by construction to an input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the algebraic form of the DPO loss and on the public-text assumption. No ad hoc constants are introduced: beta, learning rate, and batch size are inherited from standard DPO. The theoretical results hold for any epsilon > 0 and any beta > 0, so the only free parameters are training hyperparameters. The plausibility of the utility bound depends on the measured low effective rank of preference-axis gradients, which is an empirical observation, not an axiom.

free parameters (3)
  • beta (DPO temperature) = 0.1
    Standard DPO hyperparameter from Rafailov et al. [61], set to 0.1 in all experiments. The privacy mechanism is valid for any beta > 0, but the utility concentration bound (Theorem 4.2) scales linearly with beta.
  • learning rate = 5e-7
    Standard DPO learning rate from [61]. Not part of the privacy derivation.
  • batch size = 64
    Standard DPO batch size from [61]. Affects the concentration bound in Theorem 4.2 but is not fitted to the privacy mechanism.
assumptions (5)
  • domain assumption Exact algebraic form of the DPO objective and its gradient (Eq. 3, Section 2.1.2).
    The preference-axis property and the privacy mechanism rely on the precise structure of the DPO loss. If the loss were changed, the one-dimensional gradient difference would not hold.
  • domain assumption The reference model pi_ref is fixed and not updated during DPO training.
    The gradient formula in Section 4.2 treats pi_ref as constant. Standard in DPO.
  • domain assumption Bradley-Terry preference model underlies the DPO loss.
    The DPO objective is derived from the Bradley-Terry model in [61]. The privacy proof depends on the sigmoid form of the loss.
  • standard math Randomness in each mechanism invocation is independent and the adversary cannot access the random bits.
    Required by Proposition 3.2 (transformation invariance) and Proposition 3.3 (sequential composition). Standard in differential privacy.
  • domain assumption Prompts and candidate responses in DPO datasets are public or non-sensitive.
    The preference-privacy notion assumes the adversary already knows x, y_w, and y_l (Section 3.2). This is the weakest load-bearing premise for applicability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Private Direct Preference Optimization for LLM Alignment." pith.science (2026). https://pith.science/paper/7WH4LKBM

@misc{pith2026260805040,
  author       = {Pith},
  title        = {Pith review of: Private Direct Preference Optimization for LLM Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WH4LKBM}},
  note         = {Machine review of arXiv:2608.05040}
}
read the original abstract

Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.

Figures

Figures reproduced from arXiv: 2608.05040 by the authors.

Figure 1
Figure 1. Illustration of the privacy leakage surface. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Training dynamics on Anthropic-HH for Qwen2.5-3B-Instruct with privacy budget [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Varying privacy budgets (𝜖) on Anthropic-HH for Qwen2.5-3B-Instruct. The x-axis denotes the privacy budget [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Win rate and training time across model sizes. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

96 extracted references · 58 canonical work pages

  1. [1]

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In CCS. 308–318

    2016

  2. [2]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauff- mann, et al. 2024. Phi-4 technical report.arXiv preprint arXiv:2412.08905(2024)

    arXiv 2024

  3. [3]

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic Dimen- sionality Explains the Effectiveness of Language Model Fine-Tuning. InACL. 7319–7328

    2021

  4. [4]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. InAISTATS. 4447– 4455

    2024

  5. [5]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862(2022)

    arXiv 2022

  6. [6]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073(2022)

    arXiv 2022

  7. [7]

    Borja Balle and Yu-Xiang Wang. 2018. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. InICML. 394–403

    2018

  8. [8]

    Ergute Bao, Yangfan Jiang, Fei Wei, Xiaokui Xiao, Zitao Li, Yaliang Li, and Bolin Ding. 2025. Unlocking the Power of Differentially Private Zeroth-order Optimization for Fine-tuning LLMs. InUSENIX Security. 1569–1588

    2025

Show all 96 references
  1. [9]

    Federico Barbero, Xiangming Gu, Christopher A Choquette-Choo, Chawin Sitawarin, Matthew Jagielski, Itay Yona, Petar Veličković, Ilia Shumailov, and Jamie Hayes. 2025. Extracting alignment data in open models.arXiv preprint arXiv:2510.18554(2025)

    2025

  2. [10]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al . 2023. Pythia: A suite for analyzing large language models across training and scaling. In...

    2023

  3. [11]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika39, 3/4 (1952), 324–345

    1952

  4. [12]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al

  5. [13]

    Róbert Istvan Busa-Fekete, Andres Munoz Medina, Umar Syed, and Sergei Vas- silvitskii. 2023. Label differential privacy and private training data release. In ICML. 3233–3251

    2023

  6. [14]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. InICLR

    2023

  7. [15]

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. InUSENIX Security. 267–284

    2019

  8. [16]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. InUSENIX Security. 2633–2650

    2021

  9. [17]

    Keyu Chen, Hao Tang, Qinglin Liu, and Yizhao Xu. 2025. Improved Algorithms for Differentially Private Language Model Alignment.arXiv preprint arXiv:2505.08849 (2025)

    2025 arXiv

  10. [18]

    Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. InNeurIPS. 4302–4310

    2017

  11. [19]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic...

    2025 arXiv

  12. [20]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2024. ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback. InInternational Conference on Machine Learning. PMLR, 9722–9744

    2024

  13. [21]

    Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. 2022. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650(2022)

    2022 arXiv

  14. [22]

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter- efficient fine-tuning of large-scale pre-trained language models.Nature machine intelligence5, 3 (2023), 220–235

    2023

  15. [23]

    Irit Dinur and Kobbi Nissim. 2003. Revealing information while preserving privacy. InPODS. 202–210

    2003

  16. [24]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  17. [25]

    John C Duchi, Michael I Jordan, and Martin J Wainwright. 2018. Minimax optimal procedures for locally private estimation.J. Amer. Statist. Assoc.113, 521 (2018), 182–201

    2018

  18. [26]

    Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. 2006. Our data, ourselves: Privacy via distributed noise generation. InEUROCRYPT. 486–503

    2006

  19. [27]

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Cali- brating noise to sensitivity in private data analysis. InTheory of cryptography conference. 265–284

    2006

  20. [28]

    Cynthia Dwork and Kobbi Nissim. 2004. Privacy-preserving datamining on vertically partitioned databases. InCRYPTO. 528–544

    2004

  21. [29]

    Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. 2014. Rappor: Random- ized aggregatable privacy-preserving ordinal response. InCCS. 1054–1067

    2014

  22. [30]

    Hossein Esfandiari, Vahab Mirrokni, Umar Syed, and Sergei Vassilvitskii. 2022. Label differential privacy via clustering. InAISTATS. 7055–7075

    2022

  23. [31]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Model alignment as prospect theoretic optimization. InICML. 12634– 12651

    2024

  24. [32]

    Qizhang Feng, Siva Rajesh Kasa, SANTHOSH KUMAR KASA, Hyokun Yun, Choon Hui Teo, and Sravan Babu Bodapati. 2025. Exposing Privacy Gaps: Mem- bership Inference Attack on Preference Data for LLM Alignment. InAISTATS. 5221–5229

    2025

  25. [33]

    Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. 2021. Deep learning with label differential privacy. InNeurIPS. 27131– 27145

    2021

  26. [34]

    Anmol Goel, Yaxi Hu, Iryna Gurevych, and Amartya Sanyal. 2025. Differentially Private Steering for Large Language Model Alignment. InICLR

    2025

  27. [35]

    David Gross. 2011. Recovering low-rank matrices from few coefficients in any basis.IEEE Transactions on Information Theory57, 3 (2011), 1548–1566

    2011

  28. [36]

    Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A Choquette-Choo, Katherine Lee, and A Feder Cooper

  29. [37]

    Yi He and Xingyu Zhou. 2024. On the Sample Complexity of Differentially Private Policy Optimization. InNeurIPS

    2024

  30. [38]

    Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, and Giulia Fanti. 2025. Private Federated Learning using Preference-Optimized Synthetic Data. InICML

    2025

  31. [39]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InICLR

    2022

  32. [40]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  33. [41]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts.arXiv preprint arXiv:2401.04088(2024)

    2024 arXiv

  34. [42]

    Yangfan Jiang, Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi

  35. [43]

    Daniel Kifer and Bing-Rong Lin. 2010. Towards an axiomatization of statistical privacy and utility. InPODS. 147–158

    2010

  36. [44]

    Daniel Kifer and Bing-Rong Lin. 2012. An axiomatic view of statistical privacy and utility.Journal of Privacy and Confidentiality4, 1 (2012)

    2012

  37. [45]

    Daniel Kifer and Ashwin Machanavajjhala. 2014. Pufferfish: A framework for mathematical privacy definitions.TODS39, 1 (2014), 1–36

    2014

  38. [46]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. InICLR

    2015

  39. [47]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In SOSP. 611–626

    2023

  40. [48]

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the Intrinsic Dimension of Objective Landscapes. InICLR

    2018

  41. [49]

    Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. 2022. Large Language Models Can Be Strong Differentially Private Learners. InICLR

    2022

  42. [50]

    Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov

  43. [51]

    Mani Malek, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramèr. 2021. Antipodes of Label Differential Privacy: PATE and ALIBI. InNeurIPS

    2021

  44. [52]

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. InNeurIPS. 53038–53075.14 Private Direct Preference Optimization for LLM Alignment

    2023

  45. [53]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. SimPO: simple preference optimization with a reference-free reward. InNeurIPS. 124198–124235

    2024

  46. [54]

    Ilya Mironov. 2017. Rényi differential privacy. InCSF. 263–275

    2017

  47. [55]

    Ilya Mironov, Kunal Talwar, and Li Zhang. 2019. Rényi Differential Privacy of the Sampled Gaussian Mechanism.arXiv preprint(2019)

    2019

  48. [56]

    Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagiel- ski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr, and Katherine Lee. 2025. Scalable extraction of training data from aligned, production language models. InICLR

    2025

  49. [57]

    OpenAI. 2025. GPT-5 System Card. https://cdn.openai.com/gpt-5-system-card. pdf

    2025

  50. [58]

    OpenAI. 2025. OpenAI Platform. https://platform.openai.com/docs/guides/direct- preference-optimization

    2025

  51. [59]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

  52. [60]

    Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. 2018. Scalable Private Learning with PATE. In ICLR

    2018

  53. [61]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. InNeurIPS. 53728–53741

    2023

  54. [62]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

  55. [63]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  56. [64]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. InIEEE S&P. 3–18

    2017

  57. [65]

    Amer Sinha, Thomas Mesnard, Ryan McKenna, Daogao Liu, Christopher A Choquette-Choo, Yangsibo Huang, Da Yu, George Kaissis, Zachary Charles, Ruibo Liu, et al . 2025. VaultGemma: A Differentially Private Gemma Model. arXiv preprint arXiv:2510.15001(2025)

    2025

  58. [66]

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. InNeurIPS. 3008–3021

    2020

  59. [67]

    Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mit- tal. 2025. Private Fine-tuning of Large Language Models with Zeroth-order Optimization.Transactions on Machine Learning Research(2025)

    2025

  60. [68]

    Qwen Team. 2024. Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  61. [69]

    Noel Teku, Fengwei Tian, Payel Bhattacharjee, Souradip Chakraborty, Am- rit Singh Bedi, and Ravi Tandon. 2025. Props: Progressively private self-alignment of large language models.arXiv preprint arXiv:2508.06783(2025)

    2025

  62. [70]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  63. [71]

    Florian Tramer and Dan Boneh. 2021. Differentially Private Learning Needs Better Features (or Much More Data). InICLR

    2021

  64. [72]

    Florian Tramèr, Gautam Kamath, and Nicholas Carlini. 2024. Position: consider- ations for differentially private learning with large-scale public pretraining. In ICML. 48453–48467

    2024

  65. [73]

    Ning Wang, Xiaokui Xiao, Yin Yang, Jun Zhao, Siu Cheung Hui, Hyejin Shin, Junbum Shin, and Ge Yu. 2019. Collecting and analyzing multidimensional data with local differential privacy. InICDE. 638–649

    2019

  66. [74]

    Tianhao Wang, Jeremiah Blocki, Ninghui Li, and Somesh Jha. 2017. Locally differentially private protocols for frequency estimation. InUSENIX Security 17. 729–745

    2017

  67. [75]

    Stanley L Warner. 1965. Randomized response: A survey technique for eliminat- ing evasive answer bias.J. Amer. Statist. Assoc.60, 309 (1965), 63–69

    1965

  68. [76]

    Wenqian Weng, Yi He, and Xingyu Zhou. 2025. Improved Bounds for Private and Robust Alignment.arXiv preprint arXiv:2512.23816(2025)

    2025 arXiv

  69. [77]

    Fan Wu, Huseyin A Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. 2024. Privately aligning language models with rein- forcement learning. InICLR

    2024

  70. [78]

    Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. 2024. Privacy-preserving instructions for aligning large language models. InICML. 57480–57506

    2024

  71. [79]

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al

  72. [80]

    Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, and Di Wang. 2025. Towards user-level private reinforcement learning with human feedback.arXiv preprint arXiv:2502.17515(2025)

    2025 arXiv

  73. [81]

    Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. 2024. DPZero: private fine-tuning of language models without back- propagation. InICML. 59210–59246

    2024

  74. [82]

    Yizhou Zhang, Kishan Panaganti, Laixi Shi, Juba Ziani, and Adam Wierman

  75. [83]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.PVLDB16, 12 (2023), 3848–3860

    2023

  76. [84]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InNeurIPS. 46595–46623

    2023

  77. [85]

    Xingyu Zhou, Yulian Wu, Wenqian Weng, and Francesco Orabona. 2025. Square 𝜒PO: Differentially Private and Robust𝜒2-Preference Optimization in Offline Direct Alignment. InICML. 79389–79413

    2025

  78. [86]

    Yuqing Zhu and Yu-Xiang Wang. 2019. Poisson subsampled rényi differential privacy. InICML. 7634–7642

    2019

  79. [87]

    Differentially Private Fine-tuning of Language Models. InICLR

  80. [91]

    KL-regularization Itself is Differentially Private in Bandits and RLHF.arXiv preprint arXiv:2505.18407(2025)

    2025

  81. [96]

    𝑚∑︁ 𝑖=1 𝑍𝑖 ≤ √︄ 2𝑉log 2 𝛾+ 𝑀 3 log 2 𝛾 # ≥1−𝛾. Proof. The standard Bernstein’s inequality states that for any 𝑡>0, it holds that Pr

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593(2019). A First-Cut Solutions We present two DP-based first-cut ...

    2019 arXiv

  82. [2015]

    Trust region policy optimization. InICML. 1889–1897

  83. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  84. [2021]

    Towards understanding and mitigating social biases in language models. InICML. 6565–6576

  85. [2022]

    In NeurIPS

    Training language models to follow instructions with human feedback. In NeurIPS. 27730–27744

  86. [2023]

    arXiv preprint arXiv:2303.12712(2023)

    Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712(2023)

    2023 arXiv

  87. [2024]

    InIEEE S&P

    Protecting label distribution in cross-silo federated learning. InIEEE S&P. 4828–4847

  88. [2025]

    Measuring memorization in language models via probabilistic extraction. InNAACL. 9266–9291

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.