pith. sign in

arxiv: 2605.20521 · v1 · pith:CQEWXJEFnew · submitted 2026-05-19 · 💻 cs.LG · cs.CR

An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees

Pith reviewed 2026-05-21 07:02 UTC · model grok-4.3

classification 💻 cs.LG cs.CR
keywords differential privacyfine-tuningexponential mechanismquadratic approximationmachine learningprivacy guaranteesrandom projectionmultivariate normal
0
0 comments X

The pith

A local quadratic approximation to the loss enables the exponential mechanism to sample fine-tuned model parameters exactly from a multivariate normal distribution while guaranteeing differential privacy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a randomized algorithm for fine-tuning pretrained machine learning models on small sensitive datasets while ensuring differential privacy. It constructs a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. Theoretical privacy guarantees, sensitivity bounds, and accuracy estimations are established for the method. A random-projection strategy is introduced to scale the approach to high-dimensional models, with experiments on MNIST and MIMIC showing competitive performance.

Core claim

By constructing a utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset, the exponential mechanism admits exact sampling from a multivariate normal distribution in closed form, for which privacy guarantees and accuracy estimates can be derived directly.

What carries the argument

The utility function formed by a local quadratic approximation of the pretrained model's loss combined with the new dataset, enabling closed-form multivariate normal sampling in the exponential mechanism.

Load-bearing premise

A local quadratic approximation of the pretrained model loss combined with the new dataset yields a utility function whose sensitivity can be bounded tightly enough to deliver meaningful privacy while preserving useful accuracy.

What would settle it

Observe whether the method maintains its claimed accuracy when applied to a model whose loss surface is known to be strongly non-quadratic away from the pretrained parameters, such as a deep network with many layers trained on complex data.

Figures

Figures reproduced from arXiv: 2605.20521 by Alberto Bocchinfuso, Christopher Stanley, Hoang Tran, Jiayi Wang, Jorge Ramirez, M. Paul Laiu.

Figure 1
Figure 1. Figure 1: shows the fine-tuning performance of ExpM￾Quad on the sinusoidal regression with fixed projected dimension p˜ = 20 and varied radius R. The best performance of ExpM-Quad is achieved at moderate R (0.1, 0.15) where the method yields lower loss, passing the zero-shot loss value at ε = 1 and approaching the non-private SGD baseline as ε increases. On the other hand, too small a radius (R = 0.01) over-restrict… view at source ↗
Figure 2
Figure 2. Figure 2: shows fine-tuning performance of ExpM-Quad with a fixed radius (R = 0.1) and varied projected dimension p˜. All variants improve as ε increases and converge toward the non-private SGD baseline at large ε. As expected, larger projected dimensions (p˜ = 20, 40) outperform smaller ones (p˜ = 5, 10), since a higher-dimensional subspace better ap￾proximates the utility optimizer and imposes less restriction on … view at source ↗
Figure 3
Figure 3. Figure 3: FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
read the original abstract

Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information. In this work, we develop a randomized algorithm based on the exponential mechanism for fine-tuning while ensuring differential privacy. Our key idea is to construct a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. We establish theoretical privacy guarantees, sensitivity bounds, and accuracy estimations for our method. We further introduce a random-projection strategy that makes the approach scalable to high-dimensional models. Numerical experiments on the MNIST benchmark and the MIMIC clinical dataset demonstrate competitive performance against existing differentially private fine-tuning techniques.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes a differentially private fine-tuning algorithm for pretrained ML models based on the exponential mechanism. A utility function is defined by combining a local quadratic Taylor approximation of the pretrained loss around the pretrained parameters with a term derived from the new dataset; the resulting quadratic utility permits exact sampling from a multivariate normal in closed form. Theoretical privacy guarantees via explicit sensitivity bounds on the utility, accuracy estimates, and a random-projection technique for high-dimensional scalability are presented. Experiments on MNIST and the MIMIC clinical dataset report competitive accuracy relative to existing DP fine-tuning baselines.

Significance. If the sensitivity bounds prove tight and the quadratic approximation remains faithful, the method supplies an efficient, closed-form alternative to noisy-gradient DP fine-tuning that avoids iterative optimization while preserving exact sampling. The exact MVN sampling and random-projection scalability are clear technical strengths that could improve reproducibility and applicability to high-dimensional models. Practical impact, however, rests on whether the derived Δ yields useful accuracy at meaningful privacy levels (ε, δ), which the current analysis leaves open.

major comments (3)
  1. [§3, Eq. (7)] §3 (Utility Construction) and Eq. (7): the sensitivity bound Δ on |u(D_new, θ) − u(D_new′, θ)| is claimed to be independent of the largest Hessian eigenvalue of the pretrained loss, yet the quadratic term explicitly involves this Hessian; the derivation therefore appears to require an additional uniform bound on curvature or gradient norms that is not stated or verified, risking a loose Δ that forces either large ε or degraded utility.
  2. [§5.3] §5.3 (Random Projection): the projection matrix is introduced post-hoc to reduce dimensionality, but no analysis is given of how the projection error affects either the exact quadratic form required for MVN sampling or the sensitivity bound itself; if the projected utility deviates from quadratic, the closed-form sampling claim no longer holds and the privacy guarantee must be re-derived.
  3. [§6, Table 2] §6 (Experiments), Table 2: accuracy is reported as competitive, yet the effective noise scale (determined by ε/Δ) and the numerical value of the sensitivity bound Δ are not tabulated; without these quantities it is impossible to judge whether the observed accuracy is achieved at a privacy level that is meaningfully stronger than the baselines.
minor comments (2)
  1. Notation for the utility function should explicitly separate the pretrained quadratic term from the new-data linear/quadratic term to avoid reader confusion when tracking sensitivity contributions.
  2. The abstract states 'exact sampling from a multivariate normal distribution in closed form,' but the manuscript never writes the explicit mean and covariance of that normal; adding this expression would clarify the sampling procedure.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their careful reading and constructive comments on our manuscript. We address each major comment point by point below, providing clarifications and committing to revisions where appropriate to strengthen the presentation.

read point-by-point responses
  1. Referee: [§3, Eq. (7)] §3 (Utility Construction) and Eq. (7): the sensitivity bound Δ on |u(D_new, θ) − u(D_new′, θ)| is claimed to be independent of the largest Hessian eigenvalue of the pretrained loss, yet the quadratic term explicitly involves this Hessian; the derivation therefore appears to require an additional uniform bound on curvature or gradient norms that is not stated or verified, risking a loose Δ that forces either large ε or degraded utility.

    Authors: We appreciate the referee highlighting this potential source of confusion. The quadratic approximation is constructed exclusively from the pretrained loss function evaluated at the pretrained parameters and is therefore identical for any choice of D_new or D_new′. Consequently, when forming the difference |u(D_new, θ) − u(D_new′, θ)| the two quadratic terms cancel exactly, and the resulting sensitivity bound Δ depends only on the dataset-dependent linear term. This cancellation is implicit in the derivation of Eq. (7) but was not stated explicitly. In the revised manuscript we will add a short remark immediately after Eq. (7) that makes the cancellation explicit and confirms that no additional uniform bound on the Hessian eigenvalues is required for the sensitivity result. revision: yes

  2. Referee: [§5.3] §5.3 (Random Projection): the projection matrix is introduced post-hoc to reduce dimensionality, but no analysis is given of how the projection error affects either the exact quadratic form required for MVN sampling or the sensitivity bound itself; if the projected utility deviates from quadratic, the closed-form sampling claim no longer holds and the privacy guarantee must be re-derived.

    Authors: We agree that a quantitative treatment of the projection error is necessary for rigor. In the revised version we will expand §5.3 with a new lemma that bounds the deviation of the projected utility from the original quadratic form using the Johnson-Lindenstrauss lemma. We will show that the deviation can be absorbed into a slightly inflated sensitivity bound Δ′ = Δ + ε_proj, where ε_proj is an explicit function of the target dimension and failure probability. The privacy analysis will be updated to use Δ′, and we will note that the sampling distribution remains exactly multivariate normal (with the adjusted covariance) provided the projection is applied before forming the quadratic utility. This preserves the closed-form sampling property while making the privacy guarantee fully rigorous. revision: yes

  3. Referee: [§6, Table 2] §6 (Experiments), Table 2: accuracy is reported as competitive, yet the effective noise scale (determined by ε/Δ) and the numerical value of the sensitivity bound Δ are not tabulated; without these quantities it is impossible to judge whether the observed accuracy is achieved at a privacy level that is meaningfully stronger than the baselines.

    Authors: This observation is correct and improves the interpretability of the experimental results. We will augment Table 2 with three additional columns reporting, for each method and dataset: (i) the computed sensitivity bound Δ, (ii) the privacy parameter ε used, and (iii) the resulting effective noise scale ε/Δ. A short paragraph will be added to §6 explaining how Δ was evaluated numerically from the dataset-dependent term. These additions will allow direct comparison of the privacy-utility operating points with the baselines. revision: yes

Circularity Check

0 steps flagged

No significant circularity; standard DP exponential mechanism applied to a constructed quadratic utility

full rationale

The paper defines a utility u(θ) explicitly as a local quadratic Taylor approximation of the pretrained loss plus a term from the new dataset D_new. The exponential mechanism then yields an MVN because exp(ε u / 2Δ) is Gaussian when u is quadratic; this is a direct mathematical consequence of the definition, not a reduction of a claimed prediction back to fitted inputs. Privacy guarantees rest on an explicit sensitivity bound Δ for neighboring datasets, which is derived from the quadratic form rather than fitted or self-cited as a uniqueness theorem. No self-citation load-bearing steps, no ansatz smuggled via prior work, and no renaming of known results as new derivations. The construction is self-contained against external DP primitives and verifiable by direct computation of the Gaussian parameters.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method rests on standard differential privacy theory and local quadratic approximations common in optimization; no new free parameters or invented entities are introduced in the abstract.

axioms (1)
  • standard math The exponential mechanism provides differential privacy when the utility function has bounded sensitivity
    Invoked to establish the privacy guarantees of the proposed mechanism.

pith-pipeline@v0.9.0 · 5682 in / 1070 out tokens · 30127 ms · 2026-05-21T07:02:29.860622+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages

  1. [1]

    Calibrating noise to sensitivity in private data analysis,

    C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” inTheory of Cryptography, S. Halevi and T. Rabin, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 265–284

  2. [2]

    Differential privacy: A survey of results,

    C. Dwork, “Differential privacy: A survey of results,” inTheory and Applications of Models of Computation, M. Agrawal, D. Du, Z. Duan, and A. Li, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 1–19

  3. [3]

    Differentially private empir- ical risk minimization with input perturbation,

    K. Fukuchi, Q. K. Tran, and J. Sakuma, “Differentially private empir- ical risk minimization with input perturbation,” inDiscovery Science, A. Yamamoto, T. Kida, T. Uno, and T. Kuboyama, Eds. Cham: Springer International Publishing, 2017, pp. 82–90

  4. [4]

    Certified robustness to adversarial examples with differential privacy,

    M. L ´ecuyer, V . Atlidakis, R. Geambasu, D. Hsu, and S. Jana, “Certified robustness to adversarial examples with differential privacy,” 05 2019, pp. 656–672

  5. [5]

    Heterogeneous gaussian mechanism: Preserving differential privacy in deep learning with provable robustness,

    N. Phan, M. N. Vu, Y . Liu, R. Jin, D. Dou, X. Wu, and M. T. Thai, “Heterogeneous gaussian mechanism: Preserving differential privacy in deep learning with provable robustness,” inProceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 7 ...

  6. [6]

    Differential privacy preservation for deep auto-encoders: an application of human behavior prediction,

    N. Phan, Y . Wang, X. Wu, and D. Dou, “Differential privacy preservation for deep auto-encoders: an application of human behavior prediction,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Feb. 2016. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/10165

  7. [7]

    Preserving differential privacy in convolutional deep belief networks,

    N. Phan, X. Wu, and D. Dou, “Preserving differential privacy in convolutional deep belief networks,”Machine Learning, vol. 106, no. 9-10, pp. 1681–1704, Oct. 2017, publisher Copyright: © 2017, The Author(s)

  8. [8]

    Adaptive laplace mechanism: Differential privacy preservation in deep learning,

    N. Phan, X. Wu, H. Hu, and D. Dou, “Adaptive laplace mechanism: Differential privacy preservation in deep learning,”2017 IEEE International Conference on Data Mining (ICDM), pp. 385–394, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:1567787

  9. [9]

    Towards practical differentially private convex optimization,

    R. Iyengar, J. P. Near, D. X. Song, O. Thakkar, A. Thakurta, and L. Wang, “Towards practical differentially private convex optimization,”2019 IEEE Symposium on Security and Privacy (SP), pp. 299–316, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:52087126

  10. [10]

    Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

    M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” inProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 308–318. [Online]. Available: https://doi.org/10.1145/297674...

  11. [11]

    Concentrated differentially private gradient descent with adaptive per-iteration privacy budget,

    J. Lee and D. Kifer, “Concentrated differentially private gradient descent with adaptive per-iteration privacy budget,” inProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 1656–1665. [Online]. Available: https://doi.org/10.1145/3...

  12. [12]

    Do not let privacy overbill utility: Gradient embedding perturbation for private learning,

    D. Yu, H. Zhang, W. Chen, and T.-Y . Liu, “Do not let privacy overbill utility: Gradient embedding perturbation for private learning,” inInternational Conference on Learning Representations, 2021

  13. [13]

    Medical imaging deep learning with differential privacy,

    A. Ziller, D. Usynin, R. Braren, M. Makowski, D. Rueckert, and G. Kaissis, “Medical imaging deep learning with differential privacy,” Scientific Reports, vol. 11, no. 1, p. 13524, 2021

  14. [14]

    Differential privacy for deep learning in medicine,

    M. Mohammadi, M. Vejdanihemmat, M. Lotfinia, M. Rusu, D. Truhn, A. Maier, and S. T. Arasteh, “Differential privacy for deep learning in medicine,”arXiv preprint arXiv:2506.00660, 2025

  15. [15]

    Analysis of application examples of dif- ferential privacy in deep learning,

    Z. Shen and T. Zhong, “Analysis of application examples of dif- ferential privacy in deep learning,”Computational intelligence and neuroscience, vol. 2021, no. 1, p. 4244040, 2021

  16. [16]

    Mechanism design via differential privacy,

    F. McSherry and K. Talwar, “Mechanism design via differential privacy,” in48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), 2007, pp. 94–103

  17. [17]

    Data mining with differential privacy,

    A. Friedman and A. Schuster, “Data mining with differential privacy,” inProceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’10. New York, NY , USA: Association for Computing Machinery, 2010, p. 493–502. [Online]. Available: https://doi.org/10.1145/1835804.1835868

  18. [18]

    Kapralov and K

    M. Kapralov and K. Talwar,On differentially private low rank approximation, pp. 1395–1414. [Online]. Available: https://epubs. siam.org/doi/abs/10.1137/1.9781611973105.101

  19. [19]

    Differentially private hierarchical count-of-counts histograms,

    Y .-H. Kuo, C.-C. Chiu, D. Kifer, M. Hay, and A. Machanavajjhala, “Differentially private hierarchical count-of-counts histograms,”Proc. VLDB Endow., vol. 11, no. 11, p. 1509–1521, Jul. 2018. [Online]. Available: https://doi.org/10.14778/3236187.3236202

  20. [20]

    Differential privacy without sensitivity,

    K. Minami, H. Arai, I. Sato, and H. Nakagawa, “Differential privacy without sensitivity,” inProceedings of the 30th International Confer- ence on Neural Information Processing Systems, ser. NIPS’16. Red Hook, NY , USA: Curran Associates Inc., 2016, p. 964–972

  21. [21]

    Are normalizing flows the key to unlocking the exponential mechanism? a path through the accuracy-privacy ceiling constraining differentially private ml,

    R. A. Bridges, V . J. Tombs, and C. B. Stanley, “Are normalizing flows the key to unlocking the exponential mechanism? a path through the accuracy-privacy ceiling constraining differentially private ml,” 2024

  22. [22]

    The algorithmic foundations of differential privacy,

    C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,”Found. Trends Theor. Comput. Sci., vol. 9, no. 3–4, p. 211–407, Aug. 2014. [Online]. Available: https://doi.org/10.1561/ 0400000042

  23. [23]

    How to dp-fy ml: A practical tutorial to machine learning with differential privacy,

    N. Ponomareva, S. Vassilvitskii, Z. Xu, B. McMahan, A. Kurakin, and C. Zhang, “How to dp-fy ml: A practical tutorial to machine learning with differential privacy,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 5823–5824. [Online]. Ava...

  24. [24]

    Opacus: User-friendly differential privacy library in pytorch

    A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov, “Opacus: User-friendly differential privacy library in PyTorch,”arXiv preprint arXiv:2109.12298, 2021

  25. [25]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” inProceedings of the 34th International Conference on Machine Learning (ICML), 2017

  26. [26]

    Gradient-based learning applied to document recognition,

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998

  27. [27]

    The eicu collaborative research database, a freely available multi-center database for critical care research,

    T. J. Pollard, A. E. W. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi, “The eicu collaborative research database, a freely available multi-center database for critical care research,”Sci Data, vol. 5, p. 180178, 2018

  28. [28]

    Mimic-iv, a freely accessible electronic health record dataset,

    A. E. W. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, L.-w. H. Lehman, L. A. Celi, and R. G. Mark, “Mimic-iv, a freely accessible electronic health record dataset,”Sci Data, vol. 10, no. 1, 2023

  29. [29]

    Introducing the blendedicu dataset, the first harmonized, international intensive care dataset,

    M. Oliver, J. Allyn, R. Carencotte, N. Allou, and C. Ferdynus, “Introducing the blendedicu dataset, the first harmonized, international intensive care dataset,”Journal of Biomedical Informatics, vol. 146, p. 104502, 2023. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S153204642300223X

  30. [30]

    An Extensive Data Processing Pipeline for MIMIC-IV,

    M. Gupta, B. Gallamoza, N. Cutrona, P. Dhakal, R. Poulain, and R. Beheshti, “An Extensive Data Processing Pipeline for MIMIC-IV,” inProceedings of the 2nd Machine Learning for Health symposium, ser. Proceedings of Machine Learning Research, vol. 12 VOLUME , <Society logo(s) and publication title will appear here.>

  31. [31]

    PMLR, 28 Nov 2022, pp. 311–325. [Online]. Available: https://proceedings.mlr.press/v193/gupta22a.html Hoang A. Tranreceived the M.S. degree in Mathematics from the Univer- sit´e d’Orl´eans, Orl ´eans, France, in 2008, and the Ph.D. degree in Applied Mathematics from the University of Pittsburgh, Pittsburgh, PA, USA, in

  32. [32]

    His research interests include compressed sensing, machine learning, high-dimensional approximations and numerical solution of partial differential equations

    He is currently a mathematician with Data Analysis and Machine Learning Group, Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA. His research interests include compressed sensing, machine learning, high-dimensional approximations and numerical solution of partial differential equations. Jorge Ramirezis a Colombi...

  33. [33]

    Her research interests include federated learning, differential privacy, synthetic data generation, and distributed optimization

    She is currently a Postdoctoral Researcher at Oak Ridge National Laboratory. Her research interests include federated learning, differential privacy, synthetic data generation, and distributed optimization. Alberto Bocchinfusoreceived his BS in Computer Engineering from University of Calabria (Italy) in 2016, his MS in Automation and Control Engineering f...