REVIEW 6 minor 45 references
In Defense of Using Worst-case Privacy Disclosure as Privacy Evaluation Metric of Voice Anonymization
T0 review · 0 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Voice anonymization should be judged by worst-case log-likelihood leakage, not equal error rate, because EER=50% can coexist with large per-speaker information disclosure.
desk verdict A sound, clearly-written case that EER is the wrong privacy metric; the rank-to-LLR unification is the only real caveat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the log-likelihood ratio $\mathrm{LLR}(s) = \log \frac{p(s|H_Y)}{p(s|H_N)}$, where $H_Y$ is the hypothesis that probe and enrollment match and $H_N$ that they do not. Under the threat model, the defender can only influence this term; perfect secrecy is $\mathrm{LLR}(s)=0$ for every $s$, and $\epsilon_{\max}$ measures how far the worst probe is from that ideal. The paper also derives a rank-based LLR identity, converting a histogram of target-speaker ranks $\gamma_k$ into log-likelihoods, then uses it to show that the rank metric's zero-leakage condition $\gamma_k = 1/N$ is identical to perfect secrecy.
What would settle it
Run a numerical search over pairs of score distributions $p(s|H_Y)$ and $p(s|H_N)$ that satisfy the LLR-of-LLR identity $r = \log \frac{p(r|H_Y)}{p(r|H_N)}$ and ask whether any such pair yields EER=50% while some $r$ differs from zero. The paper's proof says no such pair exists; finding one would collapse the claim that EER=50% is ill-defined for ideal LLRs. Alternatively, simulate rank data with non-exchangeable remaining speakers and check whether Eq. (35) reproduces the oracle LLR.
Extended reading notes
Core claim
The paper's central claim is that perfect secrecy for voice anonymization means $\mathrm{LLR}(s)=0$ for every probe utterance, so the appropriate evaluation metric is $\epsilon_{\max} = \max_s |\mathrm{LLR}(s)|$ rather than EER. An 'ideal' system in EER terms may still leak substantial information about individual speakers. When scores are ideal LLRs, EER=50% is not even definable; when raw scores are used, EER=50% is necessary but not sufficient for perfect secrecy, as a Gaussian counterexample with equal means and different variances demonstrates. The paper further claims that the rank-based disclosure metric can be expressed as $\mathrm{LLR}(k) = \log\left(\gamma_k (N-1)/(1-\gamma_k)\right)$, making zero leakage in rank terms exactly equivalent to perfect secrecy.
Load-bearing premise
The load-bearing premise is that once the target speaker's rank is fixed, every other speaker is equally likely to sit in each remaining rank; if real score distributions violate this exchangeability, the rank-based leakage measure stops being a true log-likelihood ratio.
Editorial extensions
If this is right
- Evaluations that report only EER can certify anonymizers that actually leak speaker identity, so benchmarks should add a worst-case LLR leakage figure.
- The rank-based disclosure metric and the LLR-based metric share the same optimum, which means the simpler rank computation is a legitimate LLR estimator.
- Since PAV-based LLRs can cap or skew the extremes of the score range, the choice of calibration method becomes part of the privacy guarantee being assessed.
- On the VoicePrivacy challenge scores, none of the examined systems reaches perfect secrecy, and the rank order of systems depends on the LLR estimator used.
Reading between the lines
- A corollary the paper leaves implicit is that per-speaker worst-case leakage can be reported as a privacy profile, showing which speakers are most exposed instead of averaging over everyone.
- The rank-based LLR identity suggests a threshold-free calibration that avoids the tail instabilities of PAV; this could be tested as a drop-in replacement on future challenge data.
- If the exchangeability assumption behind Eq. (35) fails in real enrollment sets, rank-based LLRs will be biased; a direct simulation with non-exchangeable ranks would reveal the size of the bias.
- Because epsilon_max focuses on the single most leaked utterance, it is naturally sensitive to outliers; a practical extension would pair it with a quantile of the LLR distribution for stability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper argues that voice anonymization privacy should be evaluated by the worst-case absolute log-likelihood ratio (epsilon_max) under Shannon perfect secrecy, rather than by the Equal Error Rate (EER). The authors first formalize the attacker's Bayes-optimal decision and define perfect secrecy as LLR(s)=0 for every probe score. They then show that EER=50% is necessary but not sufficient for perfect secrecy when raw scores are used, and that EER=50% is not even well-defined for idealized LLRs. The paper also casts a recently proposed rank-based metric into the perfect-secrecy framework under an explicit exchangeability assumption, discusses pitfalls of PAV-based LLR estimation, and demonstrates the arguments on simulated data and on VoicePrivacy 2024 challenge scores.
Significance. If the results hold, the paper provides a clear theoretical justification for preferring worst-case LLR-based metrics over EER in voice anonymization evaluation, and it offers an accessible reconciliation of several existing metrics. The core counterexample in Section 4.3.3 is mathematically valid, and the proof in Section 4.3.1 that EER=50% is unattainable for non-degenerate ideal LLR distributions is sound under the stated assumptions. The manuscript is transparent about its assumptions, includes a reproducible Jupyter notebook, and demonstrates the claims on real challenge data. The main conceptual contribution is not a new metric but a rigorous clarification of the relationship between EER, rank-based metrics, and perfect secrecy, which is valuable for the community.
minor comments (6)
- [Section 4.3.1, after Eq. (24)] The sentence 'the attacker cannot find a τ_EER so that p_miss(τ_EER)=pmiss(τ_EER)' contains a typo: the second probability should be p_fa(τ_EER). Please correct this.
- [Eq. (9)] The definition epsilon_max = argmax_{s in S} |LLR(s)| is notationally inaccurate because argmax returns the score value, not the maximum absolute LLR. Please replace it with max_{s in S} |LLR(s)|, or define epsilon_max as the value of the maximum.
- [Section 5.2, Eq. (34)] The exchangeability assumption P_R(k|T_r)=1/(N-1) is explicit but its validity for real ASV score distributions is not discussed. If this assumption fails, Eq. (35) is not the true log-likelihood ratio, and the rank-based LLR should be interpreted as an approximation; a sentence noting this caveat and its practical impact would strengthen the paper.
- [Section 6.1, Table 2, Example 2] The text states that the maximum LLR magnitude in Example 2 becomes 0, but it may be worth clarifying explicitly that this artifact arises from PAV smoothing after label flipping, where every pooled bin receives posterior probability 1/2; this helps readers understand why the flipped example yields zero leakage despite separable scores.
- [Section 3.2] The suggested link to differential privacy is presented as a formal connection, but the comparison is an analogy rather than a proof of DP guarantees because the anonymization mechanism is not necessarily randomized. A caveat to this effect would prevent over-interpretation.
- [Section 8, Conclusion] The sentence 'This study is in progress' is more appropriate for a preprint than a journal submission. Please rephrase it as a statement about future work or remove it.
Circularity Check
No significant circularity: the EER-vs-perfect-secrecy argument is self-contained; the rank-based LLR caveat is an explicit modeling assumption, not a circular reduction.
full rationale
The paper's central claim, that EER=50% does not imply perfect secrecy, is derived from Bayes decision theory and the definition of the log-likelihood ratio, not from a fitted parameter or a self-citation. The Gaussian counterexample in Section 4.3.3 is an explicit construction in which the score distributions have equal means but different variances; the EER is 50% while the oracle LLR is nonzero except at two points. This is a mathematical demonstration, not a prediction forced by an input. The proof in Section 4.3.1 that EER=50% is infeasible for ideal LLRs uses the identity exp(r)p(r|H_N)=p(r|H_Y), which follows from the change-of-variables relation r=log p(s|H_Y)/p(s|H_N); this result is cited to external work by van Leeuwen and Brummer, and it is a standard property rather than an assumption tailored to the conclusion. The rank-based derivation in Section 5.2 is transparent about its key assumption in Eq. (34), namely that conditional on the target rank, the remaining N-1 non-target speakers are equally likely to occupy each other rank. That assumption may fail in practice, but it is explicitly stated and is not identical to the conclusion; the final equivalence between zero rank leakage and zero LLR is a consequence of both conditions reducing to gamma_k=1/N, which is a consistency check rather than circularity. The VoicePrivacy experiments are illustrative evaluations on released challenge scores, and the paper explicitly disclaims having solved LLR estimation, so there is no fitted-input-called-prediction pattern. Self-citations to VoicePrivacy data, the threat model, and prior challenge papers are contextual or data-source references; the theoretical arguments do not depend on them in a load-bearing way. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is merely renamed. The derivation chain is therefore self-contained and not circular.
Assumptions & free parameters
assumptions (5)
- standard math Bayes decision theory and the conditional risk decomposition in Eq. (1) are applied as the attacker's optimal decision framework.
- domain assumption The threat model assumes the attacker makes a binary matched/non-matched decision using an ASV system, with a defender who cannot alter the attacker's prior or decision costs.
- standard math For ideally calibrated scores, the LLR of an LLR is the LLR itself, so exp(r)p(r|H_N)=p(r|H_Y) (Eq. (25)).
- ad hoc to paper Given the target speaker's rank r, the remaining N-1 non-target scores are exchangeable, so each non-target speaker has equal probability 1/(N-1) of occupying any other rank (Eq. (34)).
- domain assumption Privacy-ZEBRA's LLR estimation uses PAV with Laplace succession smoothing, leading to the capped LLR in Eq. (38).
Cite this review
Pith. "Pith review of In Defense of Using Worst-case Privacy Disclosure as Privacy Evaluation Metric of Voice Anonymization." pith.science (2026). https://pith.science/paper/E7XKPVJX
@misc{pith2026260810318,
author = {Pith},
title = {Pith review of: In Defense of Using Worst-case Privacy Disclosure as Privacy Evaluation Metric of Voice Anonymization},
year = {2026},
howpublished = {\url{https://pith.science/paper/E7XKPVJX}},
note = {Machine review of arXiv:2608.10318}
}
read the original abstract
The voice anonymization community mainly uses Equal Error Rate (EER) to evaluate the performance of voice identity protection. While alternative metrics such as privacy-ZEBRA and a rank-based metric have been proposed, their underlying assumptions and differences may not be well known, especially to newcomers. This paper is motivated to fill the gap. Based on the concept of Shannon's perfect secrecy (or privacy), this paper positions itself as a defense of the privacy-ZEBRA framework. While no new metric is proposed, this paper explains how an `ideal' system in terms of EER may fail to gauge the information leakage on individual speakers in the log-likelihood ratio (LLR) space. The paper also shows how the rank-based metric can be cast into a metric that follows the same principle of perfect secrecy and how their best solutions are equivalent. Furthermore, the paper explains how the method of estimating LLRs may affect the evaluation results. These discussions are, to the best of the authors' knowledge, not explored or explained in detail in existing papers. Last but not least, the findings are demonstrated on simulated and VoicePrivacy Challenge data.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Speech signals convey varied types of information, including personally identifiable information (PII) about the speaker [1, 2]. In response to the growing legislative demands (e.g., GDPR [3]) for protecting the PII, the speech research commu- nity has put great efforts in producing related technologies, most of which are referred to as voice...
work page 2024
-
[2]
Threat Model and Perfect Secrecy Before we dive into the evaluation metrics, we need to agree on the information being protected as well as how an attacker plans to steal the information. This is referred to as the threat model, or attacker model [16]. After explaining the threat model (§ 2.1), we go through the optimal Bayes decision that the attacker ca...
-
[3]
In Defense of Privacy-ZEBRA Perfect secrecy for the defender is to drive every LLR to 0. An evaluation metric, however, takes the attacker’s view, and the question is how close the LLR is to 0 for each probe utterance. With this in mind, we recap Privacy-ZEBRA in this section and then discuss EER and other metrics in the next sections. 3.1. Measuring wors...
-
[4]
collecting scoressfor the two classes (H Y andH N), for ex- ample, using the V oicePrivacy evaluation protocol and data,
-
[5]
using a method to estimateLLR(s)for eachs,
-
[6]
and reportingϵ max on the whole evaluation set.4 Step 2 is necessary becauseLLR(s)may not be equal tos. Privacy-ZEBRA originally uses an algorithm called Pool Ad- jacent Violators (PA V) [21] to obtain the LLRs (§ 6). Another method is based on a similarity rank (§ 5). 3In cryptography literature, it is written asP(H Y|s) =P(H Y) rather than in the form o...
-
[7]
Highlighted messages We now discuss EER
EER=50% Is Not Perfect Secrecy 4.1. Highlighted messages We now discuss EER. Intuitively, an attacker who cannot link the speaker identity in anonymized utterancex (p) with the attacker-known identity in enrollmentx (r) should obtain an EER equal to 50% [4]. In § 3, we mentioned that the raw ASV scores and LLRs are not necessarily equal, which is decided ...
work page 2000
-
[8]
proposed a rank-based metric that measures PII disclosure [12]
Similarity Rank is Compatible with Perfect Secrecy Recently, B¨ackstr¨om et al. proposed a rank-based metric that measures PII disclosure [12]. The core idea is to rank the speaker similarity scores and estimate the probability of a target-speaker score being at a particular rank. Then the proba- bility can be used to measure the amount of leaked informat...
Show all 45 references
-
[9]
The attacker has to calibrate the scores into LLR-like values
Notes on LLR Estimation Methods As previous sections explain, in reality, the attacker has neither true LLRs nor true distributions of raw scores. The attacker has to calibrate the scores into LLR-like values. 6.1. Potential pitfalls when using PA V Privacy-ZEBRA uses the PA V...
2024
-
[10]
All the examples show how PA V can affect the estimation of the worst-case information leakage. 6.2. Other score calibration methods In contrast to the non-parametric PA V algorithm, a generative approach assumes parametric score distributionsp(s|H Y;θ Y) andp(s|H N;θ N). With...
-
[11]
Given the attacker ASV scores (in a semi-informed attacker setting), we pooled the female and male trials and computed EERs
Demonstration Using V oicePrivacy Data To compare EER and the maximum magnitude of LLRs on real data, we used the official scores released for the V oicePrivacy attacker challenge [31]. Given the attacker ASV scores (in a semi-informed attacker setting), we pooled the female a...
-
[12]
Conclusion This paper revisits the privacy evaluation metric used for the voice anonymization task. Motivated by the fact that quite a few metrics have been proposed but not have been fully explained in terms of technical differences and underlying assumptions, this paper star...
-
[13]
We have tried to address all the comments, but some of them cannot be reflected in the camera-ready version due to lim- ited space
Acknowledgment The authors thank the reviewer for the comments and sugges- tions. We have tried to address all the comments, but some of them cannot be reflected in the camera-ready version due to lim- ited space. This work is partially supported by JST, PRESTO Grant (JPMJPR23...
-
[14]
Generative AI Use Disclosure Generative AI was used to check grammatical errors and pre- pare the code for figure plotting
-
[15]
Preserving privacy in speaker and speech characterisation,
A. Nautsch, A. Jim ´enez, A. Treiber, J. Kolberg, C. Jasserand, E. Kindt, H. Delgado, M. Todisco, M. A. Hmani, A. Mtibaa, M. A. Abdelraheem, A. Abad, F. Teixeira, D. Matrouf, M. Gomez- Barrero, D. Petrovska-Delacr ´etaz, G. Chollet, N. Evans, T. Schneider, J.-F. Bonastre, B. R...
2019
-
[16]
Privacy in Speech Technology,
T. B ¨ackstr¨om, “Privacy in Speech Technology,”Proceedings of the IEEE, vol. 113, pp. 668–692, July 2025
2025
-
[17]
The GDPR & speech data: Reflections of legal and technology communities, first steps towards a common under- standing,
A. Nautsch, C. Jasserand, E. Kindt, M. Todisco, I. Trancoso, and N. Evans, “The GDPR & speech data: Reflections of legal and technology communities, first steps towards a common under- standing,” inProc. Interspeech, pp. 3695–3699, 2019
2019
-
[18]
Introducing the V oicePrivacy initiative,
N. Tomashenko, B. M. L. Srivastava, X. Wang, E. Vincent, A. Nautsch, J. Yamagishi, N. Evans, J. Patino, J.-F. Bonastre, P.- G. No´e, and M. Todisco, “Introducing the V oicePrivacy initiative,” inProc. Interspeech, pp. 1693–1697, ISCA, Oct. 2020
2020
-
[19]
The V oicePrivacy 2022 Challenge: Progress and Perspec- tives in V oice Anonymisation,
M. Panariello, N. Tomashenko, X. Wang, X. Miao, P. Champion, H. Nourtel, M. Todisco, N. Evans, E. Vincent, and J. Yamag- ishi, “The V oicePrivacy 2022 Challenge: Progress and Perspec- tives in V oice Anonymisation,”IEEE/ACM Transactions on Au- dio, Speech, and Language Process...
2022
-
[20]
The third V oicePrivacy challenge: Preserving emotional expres- siveness and linguistic content in voice anonymization,
N. Tomashenko, X. Miao, P. Champion, S. Meyer, M. Panariello, X. Wang, N. Evans, E. Vincent, J. Yamagishi, and M. Todisco, “The third V oicePrivacy challenge: Preserving emotional expres- siveness and linguistic content in voice anonymization,”Com- puter Speech & Language, vol...
2026
-
[21]
Speaker de- identification via voice transformation,
Q. Jin, A. R. Toth, T. Schultz, and A. W. Black, “Speaker de- identification via voice transformation,” in2009 IEEE workshop on automatic speech recognition & understanding, pp. 529–533, IEEE, 2009
2009
-
[22]
DeID-VC: Speaker De- identification via Zero-shot Pseudo V oice Conversion,
R. Yuan, Y . Wu, J. Li, and J. Kim, “DeID-VC: Speaker De- identification via Zero-shot Pseudo V oice Conversion,” inProc. Interspeech, pp. 2593–2597, 2022
2022
-
[23]
Improving speaker de-identification with functional data analysis of f0 trajectories,
L. Tavi, T. Kinnunen, and R. G. Hautam ¨aki, “Improving speaker de-identification with functional data analysis of f0 trajectories,” Speech Communication, vol. 140, pp. 1–10, 2022
2022
-
[24]
V oiceCloak: Adversarial Example Enabled V oice De-Identification with Balanced Privacy and Utility,
M. Chen, L. Lu, J. Wang, J. Yu, Y . Chen, Z. Wang, Z. Ba, F. Lin, and K. Ren, “V oiceCloak: Adversarial Example Enabled V oice De-Identification with Balanced Privacy and Utility,”Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 7, June 2023
2023
-
[25]
The privacy ZEBRA: Zero evidence biometric recognition assessment,
A. Nautsch, J. Patino, N. Tomashenko, J. Yamagishi, P.-G. No ´e, J.-F. Bonastre, M. Todisco, and N. Evans, “The privacy ZEBRA: Zero evidence biometric recognition assessment,” inProc. Inter- speech, pp. 1698–1702, 2020
2020
-
[26]
Privacy Dis- closure of Similarity Rank in Speech and Language Processing,
T. B ¨ackstr¨om, M. H. Vali, M. Nguyen, and S. Rech, “Privacy Dis- closure of Similarity Rank in Speech and Language Processing,” IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 196–205, 2026
2026
-
[27]
Legally validated evaluation framework for voice anonymiza- tion,
N. Vauquier, B. M. L. Srivastava, S. A. Hosseini, and E. Vincent, “Legally validated evaluation framework for voice anonymiza- tion,” inProc. Interspeech, pp. 3229–3233
-
[28]
V oxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks,
E. Tsaprazlis, T. Lertpetchpun, T. Feng, S. P. Karimireddy, and S. Narayanan, “V oxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks,” inProc. ICASSP, pp. 19042–19046, IEEE
-
[29]
Dwork, A
C. Dwork, A. Roth, and Others,The algorithmic foundations of differential privacy, vol. 9. Now Publishers, Inc., 2014
2014
-
[30]
Scenario of Use Scheme: Threat Modelling for Speaker Privacy Protection in the Medical Domain,
M. U. Rahman, M. Larson, L. T. Bosch, and C. Tejedor-Garc ´ıa, “Scenario of Use Scheme: Threat Modelling for Speaker Privacy Protection in the Medical Domain,” inProc. SPSC, pp. 21–25, ISCA, Sept. 2024
2024
-
[31]
Design Choices for X-Vector Based Speaker Anonymization,
B. M. L. Srivastava, N. Tomashenko, X. Wang, E. Vincent, J. Ya- magishi, M. Maouche, A. Bellet, and M. Tommasi, “Design Choices for X-Vector Based Speaker Anonymization,” inProc. Interspeech, (ISCA), pp. 1713–1717, ISCA, Oct. 2020
2020
-
[32]
R. O. Duda, P. E. Hart, and D. G. Stork,Pattern classification. New York: Wiley, 2nd ed ed., 2001
2001
-
[33]
D. R. Stinson,Cryptography: theory and practice. Chapman and Hall/CRC, 2005
2005
-
[34]
Communication theory of secrecy systems,
C. E. Shannon, “Communication theory of secrecy systems,”The Bell System Technical Journal, vol. 28, no. 4, pp. 656–715, 1949
1949
-
[35]
The PA V algorithm optimizes bi- nary proper scoring rules,
N. Brummer and J. du Preez, “The PA V algorithm optimizes bi- nary proper scoring rules,” 2013
2013
-
[36]
Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias,
S. L. Warner, “Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias,”Journal of the American Sta- tistical Association, vol. 60, pp. 63–69, Mar. 1965
1965
-
[37]
A Com- prehensive Guide to Differential Privacy: From Theory to User Expectations,
N. Karmitsa, A. Airola, T. Pahikkala, and T. Pitk ¨am¨aki, “A Com- prehensive Guide to Differential Privacy: From Theory to User Expectations,” 2025. Version Number: 3
2025
-
[38]
Differ- entially Private Speaker Anonymization,
A. S. Shamsabadi, B. M. L. Srivastava, A. Bellet, N. Vauquier, E. Vincent, M. Maouche, M. Tommasi, and N. Papernot, “Differ- entially Private Speaker Anonymization,” inProc. PETS, 2023
2023
-
[39]
The distribution of cal- ibrated likelihood-ratios in speaker recognition,
D. A. van Leeuwen and N. Br ¨ummer, “The distribution of cal- ibrated likelihood-ratios in speaker recognition,” inProc. Inter- speech, pp. 1619–1623, 2013
2013
-
[40]
Out of a hundred trials, how many errors does your speaker verifier make?,
N. Br ¨ummer, L. Ferrer, and A. Swart, “Out of a hundred trials, how many errors does your speaker verifier make?,” inProc. In- terspeech, pp. 1059–1063, 2021
2021
-
[41]
NIST 2016 speaker recognition evaluation plan,
NIST, “NIST 2016 speaker recognition evaluation plan,” 2016
2016
-
[42]
An introduction to application-independent evaluation of speaker recognition sys- tems,
D. A. Van Leeuwen and N. Br ¨ummer, “An introduction to application-independent evaluation of speaker recognition sys- tems,” inSpeaker classification I, pp. 330–353, Springer, 2007
2007
-
[43]
A comparison of lin- ear and non-linear calibrations for speaker recognition,
N. Brummer, A. Swart, and D. V . Leeuwen, “A comparison of lin- ear and non-linear calibrations for speaker recognition,” inProc. Odyssey, pp. 14–18, 2014
2014
-
[44]
The BOSARIS Toolkit: The- ory, Algorithms and Code for Surviving the New DCF,
N. Br ¨ummer and E. d. Villiers, “The BOSARIS Toolkit: The- ory, Algorithms and Code for Surviving the New DCF,” 2011. arXiv:1304.2865 [stat]
2011 arXiv
-
[45]
The first voiceprivacy attacker challenge,
N. Tomashenko, X. Miao, E. Vincent, and J. Yamagishi, “The first voiceprivacy attacker challenge,” inProc. ICASSP, pp. 1–2, IEEE, 2025
2025
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.