REVIEW 2 major objections 4 minor 41 references
Differentially private scale testing via rank transformations and percentile modifications
T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper proves that a class of privatized Siegel–Tukey rank-sum tests for comparing two groups' spreads — RPST tests — is differentially private and asymptotically normal under the null, so p-values come from a normal table even though…
desk verdict A genuinely new class of private scale tests with solid empirics, but Theorem 3.3 is overbroad and needs a Q/n restriction; fixable and worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the RPST statistic $U_1 = \sum_{i=1}^{n_1} \psi(r_i) - \frac{n_1}{n}\sum_{i=1}^{n-Q}\psi(i)$, built from center-outward working ranks in which $Q$ central observations receive rank 0. Its global sensitivity is bounded by $GS^*(U_1) = \max\{\psi(n-Q), \, \psi(n-Q)+\psi(n-Q-1)-\bar\psi_Q\}$, which sets the Laplace noise scale; its null variance $\sigma^2(n_1,n_2,\psi,Q)$ is an explicit function of the group sizes, and Lemma A.3 shows this variance is largest when the groups are balanced. Privately underestimating $|n_1-n/2|$ therefore overestimates $\sigma$, making the test conservative. Asymptotic normality is carried by the Wald–Wolfowitz permutation central limit theorem together with Lyapunov's theorem.
What would settle it
Simulate the RPST procedure under the null with $Q=n-1$ so that only the smallest combined observation keeps a nonzero rank, with $n_1=\lfloor n/2\rfloor$ and any increasing $\psi$. For large $n$ the statistic equals one of two values depending on which group contains the minimum, so $\tilde U_1/\sigma$ concentrates on a two-point set rather than approaching $N(0,1)$; this directly contradicts the theorem's claim for $Q \in \{0,\dots,n-1\}$.
Extended reading notes
Core claim
The central discovery is that the entire pipeline survives privatization. Observations are ranked center-outward a la Siegel–Tukey; the $Q$ most central ranks are truncated to zero; an increasing, non-negative transformation $\psi$ with $\psi(0)=0$ is applied; and Laplace noise with scale $GS^*(U_1)/\epsilon_U$ is added to the group-1 rank sum. The resulting statistic, divided by its conservative standard error $\sigma$, converges in distribution to $N(0,1)$ under the null whenever $\psi$ satisfies a general polynomial-growth condition, and the group-size imbalance $|n_1-n/2|$ is deliberately underestimated so that the reference variance is overestimated and the type I error stays at $\alpha$. The same construction, with sensitivity $2\psi(n-Q)$, upgrades the private signed-rank test.
Load-bearing premise
The normal-limit proof rests on the premise that after truncating central ranks the surviving rank values are spread out enough for the variance to stay comparable to the largest rank; when almost all ranks are truncated to zero this spread disappears and the limit need not hold.
Editorial extensions
If this is right
- A practitioner can run a two-sample spread test with a fixed privacy budget and read p-values from a normal table, even though the group sizes $n_1$ and $n_2$ are never disclosed.
- Under tight budgets the recommended settings are slow-growing transformations ($\tan^{-1}$, $\log$) with a large truncation fraction $q$, while under loose budgets fast-growing transformations ($\psi(r)=r$ or $r^2$) with $q\approx 0.25$ give the best power.
- The private signed-rank test gains the same truncation-and-transformation flexibility, with sensitivity $2\psi(n-Q)$ and a normal null limit.
- Because the sensitivity-to-standard-error ratio decays like $O((n\lambda(1-\lambda)\epsilon_U)^{-1/2})$, the privacy noise becomes negligible relative to the signal as $n$ grows, so the asymptotics guide tuning at large sample sizes.
- The construction also supplies new non-private rank-based scale tests as a byproduct, since the RPST family had not existed in the non-private literature.
Reading between the lines
- The asymptotic-normality claim as stated allows $Q \in \{0\} \cup [n-1]$; a corrected statement would need to keep $n-Q$ growing (equivalently, $Q/n$ bounded away from 1), because at $Q=n-1$ the standardized statistic is a two-point variable rather than normal.
- The type I error control relies on the $(\epsilon,\delta)$ private group-size estimate; a pure-$\epsilon$ version would need a different construction, since the current argument uses the $\delta$-probability event that the underestimate holds.
- The budget-allocation tradeoff could be optimized analytically from the variance and sensitivity formulas; the simulations indicate an 80/20 split toward the statistic, but the formulas make a derivation possible.
- The same center-outward ranking with depth contours could support multivariate private scale tests, with sensitivity governed by the number of depth layers truncated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the RPST class of differentially private two-sample scale tests. The procedure assigns center-outward Siegel–Tukey-style ranks, zeroes out the Q most central ranks, applies a nonnegative increasing transformation ψ, forms the group-one rank sum U1, adds Laplace noise calibrated to a sensitivity bound, and computes a p-value from a normal reference distribution whose variance is based on a privately underestimated group-size imbalance. The paper states a differential privacy guarantee (Theorem 3.2), an asymptotic null normality result (Theorem 3.3), a type I error bound (Lemma D.1), and an analogous private signed-rank extension (Section 5). A large simulation study compares RPST tests with a general test-of-tests framework and makes recommendations for q and ψ.
Significance. The paper fills a genuine gap: there is no dedicated differentially private two-sample scale test, and the proposal is nonparametric and computationally simple. The simulations are extensive, the code is publicly available, and the signed-rank extension is a useful byproduct. The derivations are built on standard external tools (Wald–Wolfowitz CLT, Lyapunov CLT, Laplace mechanism, composition), and the tuning parameters q and ψ are user choices rather than fitted quantities, so the theoretical development is not circular. However, Theorem 3.3 as stated overreaches; the asymptotic normality claim needs a restriction on Q or n-Q. If this is repaired, the paper would be a solid contribution to private nonparametric testing.
major comments (2)
- [Theorem 3.3 and Appendix A.4] Theorem 3.3 is false as stated for Q close to n. Take Q = n - 1, which is allowed because Theorem 3.3 states Q ∈ {0} ∪ [n - 1]. The working-rank vector is (ψ(1), 0, ..., 0), so U1 = ψ(1)(δ - n1/n), where δ is the indicator that the single nonzero rank falls in group 1. Under H0 and the permutation null, σ²(n1,n2,ψ,Q) = n1 n2 ψ(1)² / n² = Θ(ψ(1)²), while GS*(U1) ≥ ψ(1); hence the Laplace term Z·GS*(U1)/σ has non-vanishing scale and eU1/σ does not converge to N(0,1). The proof in Appendix A.4 asserts that Condition 3 implies Condition 4, but for this example μ_r(ψ_n,Q)/μ_2(ψ_n,Q)^{r/2} = Θ(n^{r/2 - 1}) for r > 2, so Condition 4 fails. The theorem needs an additional assumption such as n - Q → ∞ (e.g., Q/n ≤ q* < 1); the simulations use q ≤ 0.75, so this does not invalidate the empirical conclusions.
- [Lemma A.6] The rate O((nλ(1-λ))^{-1/2}) in Lemma A.6 is not correct for all ψ satisfying Condition 3. For ψ(i) = log(1+i), the normalized variance C(n,ψ) = n^{-1}Σ(ψ(i)/ψ(n-Q))² - (n^{-1}Σψ(i)/ψ(n-Q))² tends to zero like 1/log n because the normalized log-ranks concentrate near zero, so σ² = o(n ψ²(n-Q)) and GS*(U1)/σ ≍ √(log n / n), which is not O(n^{-1/2}). The proof's assertion that C(n,ψ) is bounded below is therefore false. Since Lemma A.6 is used in Lemma A.7 to make the Laplace term negligible, the authors should either state the correct rate for sub-polynomial ψ or prove directly that GS*(U1)/σ → 0 under Condition 3.
minor comments (4)
- [Section 4, parameter list] The list 'ψ(n) ∈ {tan−1(n), ln(n+1), n0.5, n, , n2}' contains a double comma after 'n'; this should be cleaned up.
- [Table 2, caption] In the caption of Table 2, 'respectfully' should read 'respectively'.
- [Theorem C.2] The definition of μ_r(yn, m) in Theorem C.2 appears to be missing a factor 1/m inside the power: it should be y_i - (1/m)Σ_{j=1}^m y_j, to be consistent with Condition 4.
- [Condition 3] The growth condition 'ψ(n) = O(n^k) for any k > 0' is ambiguous: if read literally as holding for every k > 0, it excludes ψ(x) = x², which the text says satisfies Condition 3. Please specify whether the intended reading is 'for some k > 0' or 'for every k > 0'.
Circularity Check
No circular derivation chain: asymptotic normality comes from external Wald-Wolfowitz and Lyapunov CLTs, privacy from standard DP composition, and type I error control from a conservative variance argument; the sole self-citation is a non-load-bearing bibliographic example.
full rationale
The paper's central claims do not reduce to their inputs. Theorem 3.3 is proved in Appendix A.4 by checking the Wald-Wolfowitz condition (Condition 4), using Lemma A.6 for the vanishing Laplace-noise ratio, and applying Slutsky's theorem; the variance sigma^2 is the exact finite-sample permutation variance from Theorem A.1, and the sensitivity bound in Lemma A.4 is proved by exhaustive case analysis rather than imported from the authors' prior work. No parameter is fitted inside the theory: q and psi are user choices studied by simulation, and the private group-size construction is a DP mechanism with a proven conservative direction (Lemma D.1), not an empirical fit to the data being tested. The only self-citation, Ramsay and Chenouri (2023) at the end of Section 2.3, merely illustrates that percentile modifications were used in other rank tests and carries no weight in Theorems 3.2, 3.3, 5.1, or 5.2. One non-circular correctness gap should be noted: the proof of Theorem 3.3 asserts without adequate support that Condition 3 implies Condition 4 when n-Q is bounded; the skeptic's Q=n-1 example shows this implication can fail, so Theorem 3.3 as stated may be false in that regime. That is an invalid inference or a missing hypothesis, not a circular reduction of the conclusion to an assumption, and it does not change the circularity score.
Assumptions & free parameters
free parameters (3)
- q (percentile modification proportion) =
recommended 0.5 to 0.75 for low privacy, 0.25 for high privacy
- ψ (rank transformation) =
recommended tan^{-1} or ln(r+1) for low privacy; r or r^2 for high privacy
- privacy budget split (ϵU, ϵd) =
80% to ϵU recommended (Appendix E.5)
assumptions (6)
- domain assumption Condition 1: F and G lie in the same location-scale family with equal location and are absolutely continuous
- domain assumption Condition 2: n1/n → λ ∈ (0,1)
- ad hoc to paper Condition 3: ψ continuous, strictly increasing, ψ(0)=0, ψ(n)=O(n^k) for some k > 0
- domain assumption Exchangeability of group labels under H0
- standard math Wald-Wolfowitz central limit theorem (Theorem C.2) and Lyapunov CLT (Theorem C.1)
- standard math Definitions and composition properties of differential privacy (Definitions 2.1, Propositions 2.1 and 2.2)
Cite this review
Pith. "Pith review of Differentially private scale testing via rank transformations and percentile modifications." pith.science (2026). https://pith.science/paper/HTBVZW56
@misc{pith2026250703725,
author = {Pith},
title = {Pith review of: Differentially private scale testing via rank transformations and percentile modifications},
year = {2026},
howpublished = {\url{https://pith.science/paper/HTBVZW56}},
note = {Machine review of arXiv:2507.03725}
}
read the original abstract
We develop a class of differentially private two-sample scale tests, called the rank-transformed percentile-modified Siegel--Tukey tests, or RPST tests. These RPST tests are inspired both by recent differentially private extensions of some common rank tests and some older modifications to non-private rank tests. We present the asymptotic distribution of the RPST test statistic under the null hypothesis, under a very general condition on the rank transformation. We also prove RPST tests are differentially private, and that their type I error does not exceed the given level. We uncover that the growth rate of the rank transformation presents a tradeoff between power and sensitivity. We do extensive simulations to investigate the effects of the tuning parameters and compare to a general private testing framework. Lastly, we show that our techniques can also be used to improve the differentially private signed-rank test.
Figures
Reference graph
Works this paper leans on
-
[1]
Ansari, A. R. and Bradley, R. A. (1960). Rank-sum tests for dispersions. The annals of mathematical statistics, pages 1174–1189. 4
work page 1960
-
[2]
Awan, J. and Slavkovi´ c, A. (2018). Differentially private uniformly most powerful tests for binomial data. Advances in Neural Information Processing Systems , 31. 1
work page 2018
-
[3]
Barrientos, A. F., Reiter, J. P., Machanavajjhala, A., and Chen, Y. (2019). Differen- tially private significance tests for regression coefficients. Journal of Computational and Graphical Statistics, 28(2):440–453. 1
work page 2019
-
[4]
Cai, B., Daskalakis, C., and Kamath, G. (2017). Priv’it: Private and sample efficient identity testing. In International Conference on Machine Learning, pages 635–644. PMLR. 1
work page 2017
-
[5]
Campbell, Z., Bray, A., Ritz, A., and Groce, A. (2018). Differentially private anova testing. In 2018 1st International Conference on Data Intelligence and Security (ICDIS) , pages 281–285. IEEE. 1
work page 2018
-
[6]
Chenouri, S., Small, C. G., and Farrar, T. J. (2011). Data depth-based nonparametric scale tests. Canadian Journal of Statistics , 39(2):356–369. 4
work page 2011
-
[7]
Couch, S., Kazan, Z., Shi, K., Bray, A., and Groce, A. (2018). A differentially private wilcoxon signed-rank test. arXiv preprint arXiv:1809.01635 . 1, 13, 30 14
work page Pith review arXiv 2018
-
[8]
Couch, S., Kazan, Z., Shi, K., Bray, A., and Groce, A. (2019). Differentially private nonparametric hypothesis testing. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security , pages 737–751. 1, 2, 5, 6, 7, 21, 22, 23, 24, 32
work page 2019
Show all 41 references
-
[9]
Dunsche, M., Kutta, T., and Dette, H. (2022). Multivariate mean comparison under differential privacy. In Domingo-Ferrer, J. and Laurent, M., editors, Privacy in Statistical Databases, pages 31–45, Cham. Springer International Publishing. 1
2022
-
[10]
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., and Naor, M. (2006a). Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology- EUROCRYPT 2006: 24th Annual International Conference on the Theory and Applica- tions of Cryptographic Techniqu...
2006
-
[11]
Dwork, C., McSherry, F., Nissim, K., and Smith, A. (2006b). Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryp- tography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265–284. Springer. 1, 5
2006
-
[12]
Dwork, C., McSherry, F., Nissim, K., and Smith, A. (2006c). Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference , pages 265–
2006
-
[13]
and Roth, A
Dwork, C. and Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3–4):211–407. 4
2014
-
[14]
Gastwirth, J. L. (1965). Percentile modifications of two sample rank tests. Journal of the American Statistical Association , 60(312):1127–1141. 1, 2, 4
1965
-
[15]
Kazan, Z., Shi, K., Groce, A., and Bray, A. P. (2023). The test of tests: A framework for differentially private hypothesis testing. InInternational Conference on Machine Learning, pages 16131–16151. PMLR. 1, 10
2023
-
[16]
and Schrab, A
Kim, I. and Schrab, A. (2023). Differentially private permutation tests: Applications to kernel methods. arXiv preprint arXiv:2310.19043 . 1
2023 arXiv
-
[17]
and Barrientos, A
Pe˜ na, V. and Barrientos, A. F. (2022). Differentially private hypothesis testing with the subsampled and aggregated randomized response mechanism. arXiv preprint arXiv:2208.06803. 1, 10
2022 arXiv
-
[18]
Pratt, J. W. and Gibbons, J. D. (2012). Concepts of nonparametric theory . Springer Science & Business Media. 31
2012
-
[19]
and Chenouri, S
Ramsay, K. and Chenouri, S. (2023). Robust nonparametric hypothesis tests for dif- ferences in the covariance structure of functional data. Canadian Journal of Statistics , 52(1):43–78. 4 15
2023
-
[20]
and Tukey, J
Siegel, S. and Tukey, J. W. (1960). A nonparametric sum of ranks procedure for relative spread in unpaired samples. Journal of the American statistical association , 55(291):429–
1960
-
[21]
Van der Waerden, B. (1952). Order tests for the two-sample problem and their power. In Indagationes Mathematicae (Proceedings), volume 55, pages 453–458. Elsevier BV. 1, 2
1952
-
[22]
Van der Waerden, B. (1953). Order tests for the two-sample problem and their power. In Indagationes Mathematicae (Proceedings), volume 56, page 80. Elsevier. 2
1953
-
[23]
and Wolfowitz, J
Wald, A. and Wolfowitz, J. (1944). Statistical tests based on permutations of the observations. The Annals of Mathematical Statistics , 15(4):358–372. 31 16 A Proofs A.1 A useful structural theorem We begin by stating a useful structural theorem. Theorem A.1. Let x1, x2, . . ....
1944
-
[24]
E(Sk) = k n Pn i=1 xi
-
[25]
Var(Sk) = k n Pn i=1 x2 i + 2 k(k−1) n(n−1) Pn i=1 Pn j=i+1 xixj − k n Pn i=1 xi 2 . Proof. We begin with assertion 1. Let A = ( A1, A2, . . . , Ak) be a random sample from (x1, . . . , xn). For each i ∈ {1, 2, . . . , n}, let δi = 1{xi ∈ A}. Then, E(Sk) = E kX i=1 Ai ! = E nX...
-
[26]
V attains a maximum at k = n 2
-
[27]
raw rank
V is decreasing in k − n 2 . This is what we wanted to show. A.3 The privacy of the test We now, prove the privacy of the test, which follows mainly from the following lemma: Lemma A.4. It holds that for all Q ∈ {0, . . . , n}, and all increasing, non-negative real functions ψ...
-
[30]
This means that: GS(U1) ≤ | −r(p, n) + r(q, n) − |b2||
The reduction of the raw ranks of the data points in region A2 of database x by one apiece, which decreases U1 by |b2| (as working ranks decrease away from the centre). This means that: GS(U1) ≤ | −r(p, n) + r(q, n) − |b2||. Since r(1, n) = ψ(1) ≤ ||b2| + r(p, n)| ≤r(h, n) − r...
-
[32]
The addition of the row (whose raw rank becomes q) into database x′, which increases U1 by r(q, n)
-
[34]
This means that: GS(U1) ≤ | −r(p, n) + r(q, n) + |b3| − |b2||
The reduction of the raw ranks of the data points in region A2 of database x by one apiece, which decreases U1 by |b2| (as working ranks decrease away from the centre). This means that: GS(U1) ≤ | −r(p, n) + r(q, n) + |b3| − |b2||. Since 0 ≤ ||b2 + r(p, n)| ≤r(h, n) − r(1, n) ...
-
[36]
The reduction of the raw ranks of the data points in region A2 of database x by one apiece, which decreases U1 by |b2| (as working ranks decrease away from the centre)
-
[37]
23 This means that: GS(U1) ≤ −r(p, n) − |b2| + 1 n nX i=1 ψ(i)
The removal of a row from group 1, which increases U1 by 1 n Pn i=1 ψ(i). 23 This means that: GS(U1) ≤ −r(p, n) − |b2| + 1 n nX i=1 ψ(i) . Since ψ(1) ≤ ||b2| + r(p, n)| ≤r(h, n) − r(1, n) = ψ(n) − ψ(1), and Pn i=1 ψ(i) ≤ ψ(n), we have that GS( U1) ≤ ψ(n) − ψ(1). By a similar a...
-
[38]
The removal of the row with raw rank p from database x, which (by itself) decreases U1 by r(p, n)
-
[39]
The removal of a row from group 1, which increases U1 by 1 n Pn i=1 ψ(i)
-
[40]
The reduction of the raw ranks of the data points in region A3 of database x by one apiece, which increases U1 by |b3| (as working ranks increase toward the centre)
-
[41]
raw rank
The reduction of the raw ranks of the data points in region A2 of database x by one apiece, which decreases U1 by |b2| (as working ranks decrease away from the centre). This means that: GS(U1) ≤ −r(p, n) + 1 n nX i=1 ψ(i) + |b3| − |b2| . To make further progress, we will utili...
-
[42]
A1, consisting of the rows with ranks not exceeding rj; 29
-
[43]
A2, consisting of the rows with ranks exceeding rj but with ai < a ′ j, where a ′ j is the absolute difference of row j of database w; and,
-
[44]
Before proceeding further, we will define some more notation
A3, consisting of the rows with ai > a ′ j. Before proceeding further, we will define some more notation. Let ∆ j be the change in the signed rank of row j when transitioning from v to w. In addition, for i = 1 , 2, 3, let ∆Ai be the change in the sum of the signed ranks of al...
-
[45]
∂πα ∂m is positive for all m ∈ 1, n 2
-
[46]
Let us start with item 1
πα(0; n) < πα(1; n). Let us start with item 1. We note that n 2 − m is positive for all m ∈ 1, n 2 . Consequently, ∂πα ∂m is positive whenever erf −1 (1 − α)p 6m(n − m) < 1. We may also note that m(n − m) is increasing in m for 0 < m < n/2. Hence, the left-hand side of the abo...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.