REVIEW 3 major objections 5 minor 24 references
Hush! Protecting Secrets During Model Training: An Indistinguishability Approach
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A new per-secret protection definition and an LP-based sampling algorithm that trains models with substantially lower noise than DP-SGD while bounding the posterior probability of secret reconstruction.
desk verdict A genuinely new per-secret protection definition and a promising LP-based training pipeline, but the current version has a likely factor-of-two error in the privacy calibration and a dubious scaling assumption that undercut the empirical claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
The method has two parts. First, compute for each secret a privacy budget number mu, derived from the desired prior and posterior probabilities. Second, solve a linear program that assigns each training example a weight, so that total weight assigned to each secret is capped by its budget, while keeping as much data as possible. Examples are then sampled with probabilities proportional to their weights, and DP-SGD is run with Poisson sampling and a noise level calibrated to the actual sampling probabilities. On an arXiv abstract fine-tuning task, the preprocessing reduces the required noise by about 8x and improves test loss by about 7.5% compared to running DP-SGD on the whole dataset.
The proof backbone is an f-divergence generalization of Fano's inequality (Lemma 3.2) that translates a KL divergence bound into a reconstruction-probability bound. A separate experiment pretrains Gemma-style models with small Gaussian noise and shows that extraction of training text drops sharply, supporting the idea that moderate indistinguishability can hinder reconstruction.
Extended reading notes
Core claim
The paper's central claim is that for a same-owner secret-protection setting, bounding posterior reconstruction probability per secret (Definition 3.1) can be achieved by LP-weighted Poisson sampling plus DP-SGD with KL-calibrated noise, and that this yields reductions of about 8x in noise multiplier and about 7.5% in test loss versus running DP-SGD on the full dataset. If correct, the formal guarantee is the stated bound on reconstruction probability for each secret.
Load-bearing premise
The load-bearing assumption is that the data owner can reliably specify the secret set S, the containment function E (which examples contain which secrets), and the transformation predicates T_j, and that this structure is public. Section 3.2 concedes that defining these quantities is 'perhaps the most challenging part' of using the definition and that no empirical technique is provided. If secrets cannot be identified accurately in real corpora, the formal protection applies to the wrong objects and the practical value of the method collapses. A second fragile premise is the cited dominating pair from [CGM+24] used for noise calibration.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a relaxation of differential privacy for the same-owner setting, where the goal is to bound the posterior probability that an adversary reconstructs each of a set of known secrets, with per-secret privacy budgets. It defines {p_j, r_j}-secret protection (Definition 3.1), proves a KL/f-divergence-based reconstruction bound (Lemma 3.2), and proposes an algorithm that assigns example weights via a linear program, then applies Poisson sampling with DP-SGD while calibrating the per-secret noise multiplier against a dominating pair. The empirical sections compare this pipeline with running DP-SGD on the full dataset, reporting roughly an 8x reduction in noise multiplier and about 7.5% test-loss improvement, and also present a separate study on noise and memorization in pretrained LLMs.
Significance. If the formal claims held, the definition and algorithm would be a useful contribution for settings where the model trainer also owns the data and wants to protect a small set of known secrets rather than all user-level information. Lemma 3.2 is a clean and apparently correct generalization of Fano-type reconstruction bounds, and the calibration is not circular: the KL target mu_j is fixed by the desired (p_j, r_j), while the LP constant c only shapes the sampling probabilities and does not affect the formal validity of the final guarantee. The paper is also honest about the hard open problem of specifying S, E, and T_j in Section 3.2. However, the central delivery is currently compromised by two accounting issues: the replacement-sensitivity mismatch between Definition 3.1 and the dominating pair used in Algorithm 2, and the unjustified 10x scaling assumption in Section 5.1. These issues affect whether the reported noise multipliers actually deliver the promised r_j bounds and whether the empirical claims are supported as written.
major comments (3)
- [§4, Algorithm 2 step 4; Definition 3.1] The adjacency in Definition 3.1 is replacement-based: an input differing only in secret j is obtained by replacing examples containing y_j with examples satisfying T_j(x_i, x_i')=1. Under per-example clipping to norm C, replacing one sampled example can change the batch sum by up to 2C, not C. The dominating pair used in Algorithm 2 step 4, N(\sum_{x_i \in D_j} Bern(rho_i), sigma^2)^T versus N(0, sigma^2)^T, is the standard add/remove pair for sensitivity C. Unless Theorem 1 of [CGM+24] is explicitly stated for replacement adjacency with the factor 2 absorbed into C or sigma, the binary search returns a sigma that is roughly 2x too small, and the claimed per-secret reconstruction bounds are not guaranteed. Please state the exact theorem used and, if replacement adjacency is intended, use the pair N(2\sum_{x_i \in D_j} Bern(rho_i), sigma^2)^T versus N(0, sigma^2)^T or otherwise account for the 2C sensitivity.
- [§5.1, footnote 1] The statement that scaling the batch size and dataset size by 10 is 'equivalent to reducing the noise multiplier by a factor of 10' is not justified and conflicts with standard DP-SGD accounting. In the Poisson-sampled Gaussian mechanism, the privacy loss per round depends on the sampling rate p = B/N and on the noise multiplier sigma, not on the absolute sizes of B and N; scaling both by 10 leaves p unchanged, and for a fixed number of rounds T the required sigma is unchanged. If the intention was to model the effect of a larger batch on gradient-estimate noise, that is a utility effect rather than a privacy-accounting effect and should be stated as such. As written, the noise multipliers in Figure 1c and the resulting 7.5% test-loss improvement are not supported. Please either run the experiments at the claimed scale or redo the accounting without this ad hoc reduction.
- [§3.2] Section 3.2 explicitly concedes that defining S, E, and T_j is 'perhaps the most challenging part' of using the definition and states that no empirical technique for doing so is provided. Since the experiments treat arbitrary words as secrets and set T_j identically to 1, the evaluation does not demonstrate that the framework can be applied to realistic secret structures where containment and transformation predicates must be inferred. This is a scope limitation rather than a mathematical error, but it is load-bearing for the paper's practical claim and should be stated as a prominent caveat in the abstract and conclusions rather than only in the body.
minor comments (5)
- [§4, after equation (1)] The text refers to the graph G as public, but G is not defined; it should be defined as the bipartite incidence graph of the containment function E.
- [Algorithm 2, step 4] The notation D_j is used without definition; please define D_j = E^{-1}(y_j) = {x in D : y_j in E(x)} explicitly in the algorithm or its preamble.
- [Lemma 3.2, proof] The proof is correct but compressed, especially the passage 'abusing notation to let a random variable denote a distribution'; a more explicit statement of the joint sampling process would improve readability.
- [§5.1, footnote 1] The sentence 'Since we decrease the scale of the noise, this should only make any comparisons more unfavorable for our LP-based approach' is unclear, because the preceding sentence claims an equivalence rather than an actual change; please clarify whether the sigma values in Figure 1c are the actual training values or simulated values.
- [§5.2] The pretraining memorization experiments are exploratory and are connected to Definition 3.1 only informally; please label this section as a separate empirical study rather than a validation of the proposed secret-protection definition.
Circularity Check
No significant circularity: the per-secret guarantee is calibrated from the target (p_j, r_j) and benchmarked against a fresh DP-SGD baseline, not derived from its own conclusion.
full rationale
The derivation chain is not circular. Definition 3.1 fixes target parameters (p_j, r_j), and Lemma 3.2 proves a sufficient divergence-based condition for the reconstruction bound. Algorithm 2 computes each µ_j = KL(Bern(r_j), Bern(p_j)) from the target and then chooses σ_j as the minimum noise such that the KL divergence of the dominating pair is at most µ_j; this is calibration, not a fitted prediction, and the claimed r_j bound holds by the lemma's proof for any σ satisfying the inequality. The LP constant c is tuned, but the guarantee is independent of c because σ is re-calibrated to the realized sampling probabilities. The empirical improvement is measured against DP-SGD on the full dataset, a separate algorithm, not against a re-statement of the paper's own result. The main external input is Theorem 1 of [CGM+24], a formal dominating-pair result from prior work with overlapping authorship; it is cited as a theorem and the KL quantities are computed with the open-source dp accounting library, so it does not reduce to the paper's own claim. The sensitivity-factor concern raised in the skeptic note is a potential correctness or accounting issue, not an instance of circular reasoning, because the paper explicitly defines the adjacency relation and claims a dominating pair for it; whether the pair truly dominates is a mathematical verification question, not a definitional equivalence.
Assumptions & free parameters
free parameters (1)
- LP constant c =
tuned over c=2^k, k in {-6,...,4}; best value not stated
assumptions (4)
- standard math DP-SGD with Poisson sampling has a dominating pair N(sum_i Bern(rho_i), sigma^2)^T for inputs differing only in examples containing a given secret (Theorem 1 of [CGM+24]).
- domain assumption DP-SGD's group privacy parameter is roughly linear in the group size, justifying the linear per-secret constraint in LP (1).
- domain assumption The secret structure (S, E, T_j) is known and public; adjacent datasets share it.
- ad hoc to paper Scaling the batch size and dataset size by 10 is equivalent to reducing the noise multiplier by a factor of 10 in the privacy accounting.
Cite this review
Pith. "Pith review of Hush! Protecting Secrets During Model Training: An Indistinguishability Approach." pith.science (2026). https://pith.science/paper/DVNHGSYI
@misc{pith2026250600201,
author = {Pith},
title = {Pith review of: Hush! Protecting Secrets During Model Training: An Indistinguishability Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/DVNHGSYI}},
note = {Machine review of arXiv:2506.00201}
}
read the original abstract
We consider the problem of secret protection, in which a business or organization wishes to train a model on their own data, while attempting to not leak secrets potentially contained in that data via the model. The standard method for training models to avoid memorization of secret information is via differential privacy (DP). However, DP requires a large loss in utility or a large dataset to achieve its strict privacy definition, which may be unnecessary in our setting where the data curator and data owner are the same entity. We propose an alternate definition of secret protection that instead of targeting DP, instead targets a bound on the posterior probability of secret reconstruction. We then propose and empirically evaluate an algorithm for model training with this secret protection definition. Our algorithm solves a linear program to assign weights to examples based on the desired per-secret protections, and then performs Poisson sampling using these weights. We show our algorithm significantly outperforms the baseline of running DP-SGD on the whole dataset.
Figures
Reference graph
Works this paper leans on
-
[1]
Mart \' n Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proc. of the 2016 ACM SIGSAC Conf. on Computer and Communications Security ( CCS '16) , pages 308--318, 2016
work page 2016
-
[2]
Reconstructing training data with informed adversaries
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. pages 1138--1156, 05 2022
work page 2022
-
[3]
Protection against reconstruction and its applications in private federated learning, 2019
Abhishek Bhowmick, John Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers. Protection against reconstruction and its applications in private federated learning, 2019
work page 2019
-
[4]
Private empirical risk minimization: Efficient algorithms and tight error bounds
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proc. of the 2014 IEEE 55th Annual Symp. on Foundations of Computer Science (FOCS) , pages 464--473, 2014
work page 2014
-
[5]
Clement, Matthew Bierbaum, Kevin P
Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe, and Alexander A. Alemi. On the use of arxiv as a dataset. CoRR , abs/1905.00075, 2019
arXiv 1905
-
[6]
Rachel Cummings and Deven R. Desai. The role of differential privacy in gdpr compliance. 2018
work page 2018
-
[7]
Mind the privacy unit! user-level differential privacy for language model fine-tuning
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. Mind the privacy unit! user-level differential privacy for language model fine-tuning. In First Conference on Language Modeling , 2024
work page 2024
-
[8]
Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush
Zachary Charles, Arun Ganesh, Ryan McKenna, H. Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-tuning large language models with user-level differential privacy, 2024
2024
Show all 24 references
-
[9]
Quantifying memorization across neural language models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations
-
[10]
Extracting training data from large language models
Nicholas Carlini, Florian Tram \`e r, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, \'U lfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympos...
2021
-
[11]
BERT : Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter ...
2019
-
[12]
R2t: Instance-optimal truncation for differentially private query evaluation with foreign keys
Wei Dong, Juanru Fang, Ke Yi, Yuchao Tao, and Ashwin Machanavajjhala. R2t: Instance-optimal truncation for differentially private query evaluation with foreign keys. In Proceedings of the 2022 International Conference on Management of Data , SIGMOD '22, page 759–772, New York,...
2022
-
[13]
Calibrating noise to sensitivity in private data analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC) , pages 265--284, 2006
2006
-
[14]
Google's differential privacy libraries., 2022
DP Team . Google's differential privacy libraries., 2022. https://github.com/google/differential-privacy
2022
-
[15]
It's my data too: Private ml for datasets with multi-user training examples, 2025
Arun Ganesh, Ryan McKenna, Brendan McMahan, Adam Smith, and Fan Wu. It's my data too: Private ml for datasets with multi-user training examples, 2025
2025
-
[16]
Bounding training data reconstruction in DP - SGD
Jamie Hayes, Borja Balle, and Saeed Mahloujifar. Bounding training data reconstruction in DP - SGD . In Thirty-seventh Conference on Neural Information Processing Systems , 2023
2023
-
[17]
Preventing verbatim memorization in language models gives a false sense of privacy
Daphne Ippolito, Florian Tram \`e r, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546 , 2022
-
[18]
Conservative or liberal? personalized differential privacy
Zach Jorgensen, Ting Yu, and Graham Cormode. Conservative or liberal? personalized differential privacy. volume 2015, 04 2015
2015
-
[19]
Scalable extraction of training data from (production) language models
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tram \`e r, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:23...
2023 arXiv
-
[20]
The fineweb datasets: Decanting the web for the finest text data at scale, 2024
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024
2024
-
[21]
Stochastic gradient descent with differentially private updates
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing , pages 245--248. IEEE, 2013
2013
-
[22]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...
2024
-
[23]
Individual privacy accounting for differentially private stochastic gradient descent
Da Yu, Gautam Kamath, Janardhan Kulkarni, Tie-Yan Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially private stochastic gradient descent. Transactions on Machine Learning Research , 2023
2023
-
[24]
Optimal accounting of differential privacy via characteristic function
Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statist...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.