On the Use of Penalty MCMC for Differential Privacy

We view the penalty algorithm of Ceperley and Dewing (1999), a Markov chain Monte Carlo (MCMC) algorithm for Bayesian inference, in the context of data privacy. Specifically, we study differential privacy of the penalty algorithm and advocate its use for data privacy. We show that in the simple model of independent observations the algorithm has desirable convergence and privacy properties that scale with data size. Two special cases are also investigated and privacy preserving schemes are proposed for those cases: (i) Data are distributed among several data owners who are interested in the inference of a common parameter while preserving their data privacy. (ii) The data likelihood belongs to an exponential family.