pith. sign in

arxiv: 2604.20985 · v2 · pith:ZMUTFYO7new · submitted 2026-04-22 · 💻 cs.LG · cs.AI· cs.CR· stat.ML

Differentially Private Model Merging

Qichuan Yin , Manzil Zaheer , Tian Li This is my paper

Pith reviewed 2026-05-21 08:16 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CRstat.ML
keywords differential privacymodel mergingpost-processingRényi differential privacyprivacy accountingprivate mean estimationprivacy loss distribution
0
0 comments X

The pith

Two post-processing techniques let you combine existing models trained on the same data to meet any target differential privacy level without retraining.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that given a collection of models already trained on identical data at different privacy-utility points, random selection or linear combination can produce a new model that satisfies any chosen differential privacy parameter. This matters because privacy requirements often change after deployment due to regulations or user preferences, and retraining from scratch is costly. The authors supply rigorous privacy accounting for both techniques using Rényi differential privacy and privacy loss distributions. They work out the exact tradeoffs for general problems and for the special case of private mean estimation, then compare the two methods on utility. Experiments on synthetic and real-world data with several models confirm that the approaches deliver the promised privacy guarantees in practice.

Core claim

Given models trained on the same dataset with different privacy budgets, post-processing via random selection or linear combination yields a final model that satisfies any target differential privacy parameter. The privacy of these operations is accounted for using Rényi DP and privacy loss distributions both in general settings and for private mean estimation, where the resulting utility tradeoffs are precisely characterized and compared between the two mechanisms.

What carries the argument

Random selection and linear combination as post-processing rules that merge privacy loss distributions from models trained at different privacy levels while preserving an overall Rényi DP bound.

If this is right

  • Any target privacy parameter can be achieved by post-processing from the existing set of models without new training.
  • The privacy accounting holds for general machine learning problems through Rényi DP and privacy loss distributions.
  • For private mean estimation the two mechanisms permit exact characterization of the privacy-utility frontier and direct comparison.
  • No access to the original training data is needed after the initial models are obtained.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Organizations could maintain a small library of models at graded privacy levels and quickly adapt outputs when policies shift.
  • The same post-processing logic may extend to federated or distributed settings where each model carries its own privacy constraint.
  • Adaptive selection rules that incorporate observed utility could further improve the privacy-utility curve beyond fixed random or linear merging.

Load-bearing premise

The existing models were trained on the same dataset and their privacy loss distributions can be combined via the chosen post-processing rule while preserving the overall Rényi DP bound.

What would settle it

An empirical measurement showing that the privacy loss distribution or attack success rate after random selection or linear combination exceeds the bound calculated by the Rényi DP accounting for the target privacy parameter would disprove the claims.

Figures

Figures reproduced from arXiv: 2604.20985 by Manzil Zaheer, Qichuan Yin, Tian Li.

Figure 1
Figure 1. Figure 1: Privacy/utility tradeoffs of mean estimation (δ = 10−5 ). Input models are also marked in the figure. RDP. Also, both methods achieve flexible privacy by tracing out a continuous MSE/privacy tradeoff as the target privacy level changes. Moreover, LC consistently outperforms RS, validating our theoretical arguments in Section 4.1. 8.2. Results on Real Datasets We next evaluate our method on two standard ben… view at source ↗
Figure 3
Figure 3. Figure 3: Privacy/utility tradeoffs on MNIST (δ = 10−5 ) set into two disjoint halves, Dpre and Dpriv. We first train a non-private model on Dpre using standard SGD, and then use this model as initialization for DP-SGD on Dpriv. From this initialization, we train multiple private models with different DP-SGD hyperparameters, yielding different pri￾vacy/utility tradeoffs. We then apply RS and LC to these private mode… view at source ↗
Figure 4
Figure 4. Figure 4: Privacy/utility tradeoffs of CIFAR-10 (δ = 10−5 ). methods remains broadly similar. Interestingly, in this pretraining-based setting, we also ob￾serve a phenomenon that was already present in some of our earlier experiments: the merged model may outper￾form all individual candidate models. This effect is par￾ticularly visible with pretraining, which may be because a stronger common initialization places th… view at source ↗
Figure 5
Figure 5. Figure 5: Privacy/utility tradeoffs of CIFAR-10 starting from a pretrained model (δ = 10−5 ). 9. Conclusion and Future Directions To the best of our knowledge, this is the first work that stud￾ies model merging to meet flexible privacy requirements during deployment time. We have proposed two merging strategies, based on random selection and linear combina￾tion, without any additional training steps. We provide prin… view at source ↗
Figure 6
Figure 6. Figure 6: Privacy/utility tradeoffs of merging checkpoints from same run. (a) DP parameter as π changes (b) MSE as π changes [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Random selection results for mean estimation with δ = 10−5 . (a) DP parameter as λ changes (b) MSE as λ changes [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Linear combination results for mean estimation with δ = 10−5 . 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
read the original abstract

In machine learning, privacy requirements at inference or deployment time often evolve due to changing policies, regulations, or user preferences. In this work, we aim to construct a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post-processing techniques, namely random selection and linear combination, to generate final private models satisfying any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions on general problems, as well as on private mean estimation, where we precisely characterize the privacy/utility tradeoffs and compare the two mechanisms. Empirically, we demonstrate the effectiveness of our approaches and validate our analyses on several models and both synthetic and real-world datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript proposes two post-processing techniques—random selection and linear combination—to merge models trained on the same dataset with different privacy-utility tradeoffs into a final model satisfying any target differential privacy parameter without retraining. It provides privacy accounting via Rényi DP and privacy loss distributions for general problems, plus an exact characterization and comparison for private mean estimation, with empirical validation on synthetic and real-world datasets.

Significance. If the accounting holds, the ability to tune privacy post-hoc would be useful for adapting deployed models to changing regulations or preferences. The exact characterization on private mean estimation is a strength, as it allows precise comparison of the two mechanisms and provides a concrete benchmark for privacy-utility tradeoffs.

major comments (1)
  1. [general problems accounting] In the general-problems accounting (prior to the mean-estimation section): the derivation that random selection and linear combination of PLDs from same-dataset models yields a tunable Rényi DP bound does not address the joint distribution induced by shared training data. Because every model observes the identical records, the outputs are statistically dependent; the effective PLD after mixture or weighting is not necessarily a simple functional of the marginal PLDs, which undermines the claim that arbitrary target parameters are achievable without retraining.
minor comments (2)
  1. The notation for the linear-combination weights and the random-selection probabilities is introduced without an explicit definition or table summarizing the parameters.
  2. Figure captions for the empirical results could more clearly state the number of runs and error bars used to generate the plotted curves.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful reading of the manuscript and for identifying this subtlety in the general-problems accounting. We address the concern directly below and will incorporate clarifications into the revised version.

read point-by-point responses

  1. Referee: In the general-problems accounting (prior to the mean-estimation section): the derivation that random selection and linear combination of PLDs from same-dataset models yields a tunable Rényi DP bound does not address the joint distribution induced by shared training data. Because every model observes the identical records, the outputs are statistically dependent; the effective PLD after mixture or weighting is not necessarily a simple functional of the marginal PLDs, which undermines the claim that arbitrary target parameters are achievable without retraining.

    Authors: We appreciate the referee drawing attention to the dependence arising from shared training data. Although the models are trained on identical records and are therefore statistically dependent, the privacy analysis for both post-processing techniques operates on the marginal output distributions together with the individual Rényi DP guarantees already established for each model. For random selection, the merged mechanism produces an output distribution that is a convex combination of the marginals; because the selection variable is chosen independently of the data, the Rényi divergence of the resulting mixture is bounded by a convex combination of the individual Rényi divergences (or simply by their maximum when the target is the worst-case bound). An analogous argument applies to linear combination, which can be viewed as a deterministic post-processing of the selected model. Consequently, the target privacy parameter remains tunable by the selection probability or the combination weights without retraining. We will revise the general-problems section to explicitly note the dependence, supply the short proof that the marginal bounds suffice, and confirm that the same tunable guarantees hold for the dependent case. revision: yes

Circularity Check

0 steps flagged

No circularity: privacy accounting follows from standard Rényi DP and PLD rules applied to post-processing

full rationale

The derivation applies established Rényi DP composition and privacy loss distribution properties to the random selection and linear combination post-processing rules. The accounting for general problems and private mean estimation is presented as a direct characterization of the output distribution under these rules, without fitting target privacy parameters to evaluation data or redefining quantities in terms of themselves. No load-bearing step reduces to a self-citation chain, an ansatz smuggled via prior work, or a fitted input relabeled as a prediction. The central claim that arbitrary target privacy levels are achievable via merging is supported by the derived bounds rather than assumed by construction. The paper remains self-contained against external DP benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that models share the same training dataset and that their individual privacy loss distributions can be post-processed without introducing new dependencies; no free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption Models are trained on identical data with known per-model privacy parameters that can be combined via selection or linear weights.
    Invoked when stating that post-processing generates a model satisfying any target privacy parameter.

pith-pipeline@v0.9.0 · 5670 in / 1273 out tokens · 36342 ms · 2026-05-21T08:16:47.811852+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

  1. [1]

    Mironov, I

    IEEE, 2017a. Mironov, I. R´enyi differential privacy. In2017 IEEE 30th computer security foundations symposium (CSF), pp. 263–

    work page

  2. [2]

    Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H

    IEEE, 2017b. Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H. B., Vassilvitskii, S., Chien, S., and Thakurta, A. G. How to dp-fy ml: A practical guide to machine learning with differential privacy.Journal of Artificial Intelligence Research, 77:1113–1201, 2023. Puccetti, G. and Wang, R. Extremal dependence concepts. 2015. Shejwal...

    work page arXiv 2023

  3. [3]

    and σ2 LC(λ) =λ 2σ2 1 + (1−λ) 2σ2

    work page

  4. [4]

    Next we prove εRS α =D α Pπ∥Qπ ≥ α∆2 2σ2 RS(π) +o(∆ 2)

    Thus εLC α = α∆2 2σ2 LC(λ) . Next we prove εRS α =D α Pπ∥Qπ ≥ α∆2 2σ2 RS(π) +o(∆ 2). Writepfor the density ofP π and note thatQ π’s densityq(x) =p(x−∆). Define Jχ(P) := Z R (p′(x))2 p(x) dx. Step 1.We show that Dα(Pπ∥Qπ)≥ α 2 Jχ(Pπ) ∆2 +o(∆ 2).(12) 12 Differentially Private Model Merging Consider Ψα(∆) := log R p(x)α p(x−∆) 1−α dx, so that Dα(Pπ∥Qπ) = Ψα(...

    work page 2015