REVIEW 5 major objections 4 minor 27 references
This paper argues that Equal Error Rate (EER), the standard metric for voice anonymization, hides severe privacy leaks: informed attackers can re-identify speakers at very low false-positive rates even when EER looks safe, and simple attack
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 15:42 UTC pith:EK3YD6YV
load-bearing objection The low-FPR critique of EER for voice anonymization is important and the Table 1 gap looks real, but the paper needs a careful revision before it can serve as the benchmark it claims to be. the 5 major comments →
VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central discovery is that EER is not merely an imperfect summary but an actively misleading privacy metric in the anonymized-speech setting. Across synthetic and real datasets, it shows that informed attackers who can fine-tune embedding models on anonymized data and score probes by maximum similarity to a long enrollment can achieve non-trivial true-positive rates at extremely low false-positive rates, even when their EER is nearly identical to a much weaker attacker's EER. For example, at enrollment length 60 under relaxed user privacy, EER values of 36.3% and 35.1% correspond to TPRs of 0.18% and 18.83% at FPR=1e-4, respectively. The paper formalizes this with differential-pri
What carries the argument
The central machinery is membership inference as a privacy game. An adversary must decide whether two anonymized utterances come from the same speaker (User Privacy) or share an attribute value (Attribute Privacy), and the decision rule yields a (FPR, TPR) operating point. The worst-case privacy loss is upper-bounded by a log-likelihood ratio, epsilon-hat = max(ln(TPR/FPR), ln((1-FPR)/(1-TPR))). The paper emphasizes the low-FPR regime as the normatively correct operating point, and stresses two attacker enhancements: fine-tuning the embedding model on anonymized data, and max-similarity scoring over long enrollments, which preserves the strongest match rather than averaging it away.
Load-bearing premise
The result depends on the 'Informed Attacker' premise: if real-world adversaries cannot fully access the anonymizer, generate anonymized data, fine-tune the embedding model, and use max-similarity scoring over long enrollments, the reported low-FPR leakage numbers overstate actual re-identification risk.
What would settle it
Run VoxGuard's protocol on an independent corpus and a different anonymization pipeline, computing EER and TPR at FPR=1e-4 for both average- and max-similarity attackers across enrollment lengths. If the two metrics track each other without substantial divergence, the paper's central claim that EER masks high-confidence leakage would fail.
If this is right
- Reporting EER alone can understate user-privacy leakage by orders of magnitude; benchmark reports should include TPR at FPR in {1e-2, 1e-3, 1e-4}.
- Attackers that adapt to the anonymizer (fine-tuning) and use max-similarity scoring over long enrollments (L up to 200) reveal leakage that pretrained average-scoring attackers miss.
- Anonymization pipelines like the baseline evaluated here leave speaker attributes (gender, accent) separable by simple transparent classifiers, so attribute privacy needs explicit defenses.
- VoxGuard offers a dual-metric framework—User Privacy and Attribute Privacy—with DP-style bounds, suitable as a standard benchmark for comparing anonymization methods.
Where Pith is reading between the lines
- Beyond the paper: if this divergence between EER and low-FPR leakage generalizes, past voice-anonymization challenge leaderboards that ranked systems by EER may have rewarded systems with hidden high-confidence leakages, and prior negative results in voice privacy may need revisiting.
- Beyond the paper: a concrete policy implication is that anonymization providers should report the smallest epsilon at which a specified subset of users can be re-identified, analogous to differential-privacy guarantees, rather than a single average-error number.
- Beyond the paper: the near-perfect attribute leakage suggests that defenses aimed at suppressing attribute information in embeddings—for instance, adversarial training or explicit disentanglement—could be directly benchmarked using VoxGuard's attribute-privacy tasks.
- Beyond the paper: the framework implies that evaluation protocols in speech anonymization should pre-register an attacker capability model; otherwise, results are highly sensitive to whether the adversary is allowed fine-tuning and max-similarity scoring.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that EER, the standard metric for evaluating speaker anonymization, misrepresents privacy leakage because it measures average-case attacker performance. The authors introduce VoxGuard, a framework that reframes speech privacy as membership inference and evaluates it in the low-FPR regime, with user privacy and attribute privacy as two complementary notions. They report experiments on synthetic and real speech corpora, using an informed attacker model with pretrained and fine-tuned embedding models, and average- vs max-similarity scoring. The main empirical claim is that a fine-tuned attacker using max-similarity achieves dramatically higher TPR at FPR=1e-4 (e.g., 18.8% at L=60 on ParaSpeechCaps) than average-similarity (≈0.18%) despite similar EERs. They also claim that simple logistic-regression attacks recover gender and accent with near-perfect accuracy on anonymized speech. The paper concludes that EER underestimates leakage by orders of magnitude and proposes VoxGuard as a benchmark for privacy evaluation.
Significance. If the central claim is correct, the paper makes a valuable methodological point for the speech-privacy community: reporting EER alone can hide severe worst-case leakage, and low-FPR metrics should complement or replace average-case measures. The informed-attacker threat model and the use of fine-tuned, max-scoring attackers are appropriately adversarial and go beyond the typical evaluation in VoicePrivacy challenges. The paper also introduces a clean formal distinction between user and attribute privacy, inspired by membership inference and differential privacy. The main strengths are the explicit low-FPR evaluation protocol, the use of both synthetic and real data, and the reproducible trial construction (100k balanced trials). However, the attribute-privacy section is currently unsupported by any quantitative results, and the user-privacy headline numbers lack confidence intervals and appear internally inconsistent in one place. These issues must be fixed before the central claims can be fully assessed.
major comments (5)
- [§5.2] The attribute-privacy claim is not supported by any reported numbers. The text states that transparent attacks achieve 'near-perfect separation of accent and gender' and that cosine-similarity baselines produce 'largely random results,' but there is no table, figure, accuracy, AUC, TPR/FPR, or confidence interval for any attribute experiment. Since the paper's abstract and contributions explicitly claim that attribute privacy is vulnerable, this omission is load-bearing. Please add a results table with per-dataset, per-attribute, strict/relaxed, and attacker-variant breakdowns, including standard errors or bootstrap intervals.
- [§5.1 and Table 1] The central example is internally inconsistent. §5.1 states that FT-Avg at L=60 achieves TPR≈0.18% at FPR=1e-4, but Table 1 lists FT-Avg L=60 as 0.40%; the 0.18% value appears at L=30. This discrepancy makes the FT-Max vs FT-Avg gap (18.8% vs 0.18%) non-reproducible from the paper as written. Either the table or the text is wrong; please correct and re-check all entries for consistency.
- [§4.1.3 and Table 1] No confidence intervals are reported for the low-FPR TPRs, despite the text saying 'confidence intervals are obtained via 200× bootstrap resampling over half of the available trials.' At FPR=1e-4 with 100k balanced trials, only ~5 negative trials define the threshold, so the tail estimates are highly sensitive to a few high-scoring negatives. The FT-Max vs FT-Avg separation at L=60 could be threshold noise or driven by a handful of outlier trials. Please report bootstrap CIs (or at least the number of trials defining the tail) for each cell in Table 1. Additionally, describe how the operating point was calibrated; if the same trials are used to select the threshold and evaluate TPR, the estimate can be optimistically biased. A held-out calibration split or cross-validated threshold selection is needed.
- [§3.3 and Eq. (5)] The paper calls the log-likelihood-ratio quantity an 'upper bound' on worst-case privacy loss and presents it as DP-grounded, but Eq. (5) is an empirical statistic computed from measured TPR/FPR at an operating point. No argument is made that the anonymization mechanism satisfies the (epsilon,delta) definitions in §3.1–3.2; hence the estimated epsilon is an attack-based empirical bound, not a certified DP guarantee. Moreover, delta=1e-4 is fixed but never appears in Eq. (5). Please clarify that this epsilon is an empirical attack statistic, and either derive the relation to delta or remove the DP-certification language. This does not invalidate the methodology, but it prevents overclaiming.
- [§4.1.3 and §5.1] The paper's main conclusion — that EER underestimates leakage — is established under the 'Informed Attacker' threat model, which grants the adversary full access to the anonymization system, the ability to generate arbitrary anonymized data, and fine-tuning on that data. This is a worst-case model, which is defensible for benchmarking, but the paper does not discuss when this model is realistic or how the reported low-FPR TPRs should be interpreted for weaker, real-world adversaries. Since the title and abstract make a general claim ('EER substantially underestimates leakage'), please add a paragraph explicitly situating the results as worst-case upper bounds and noting that the absolute TPR values may not transfer to settings without system access or fine-tuning capability.
minor comments (4)
- [§5.2] Typo: 'we shift to transparent attacks, We train separate classifiers' — stray capital 'W' after a comma. Also, the phrase 'the interpretation of randomness in these results, as well as the behavior observed under ATTR-PRIVACY-RELAXED, is left for future work' is vague and does not explain what was observed; either clarify or remove.
- [Table 1] The table formatting is confusing: the header says 'Each cell reports: EER (%) | epsilon | TPR@0.1%FPR (%) | TPR@0.01%FPR (%)', but some rows (e.g., PT-Avg L=1) appear to contain more than four values or a line break. Please reformat so each cell has exactly the four metrics, and ensure all entries align with their column headers.
- [§4.1.1] The description of synthetic data says 'Six speakers are selected from the Vox-Profile test set and have the probability of Vox-Profile to that desired accent more than 95% and low background noise.' Consider rephrasing for clarity and report which six speakers and how accent probability was measured.
- [§5.1] The claim 'FT–Max at L= 200 achieves TPR = 58.7% at FPR = 10−4, compared to only 1.2% for FT–Avg' uses the text values, but Table 1 shows FT-Avg L=200 TPR@0.01%FPR=0.28% and TPR@0.1%FPR=1.16%. Please state precisely which operating point (0.01% or 0.1%) is being compared.
Circularity Check
No significant circularity: the paper's main claims rest on measured TPR/FPR and standard MIA bounds, not on self-referential definitions or fitted predictions.
full rationale
The paper's derivation chain is empirical rather than definitional. The central claims—EER underreports leakage, fine-tuned max-similarity attackers are stronger, and speaker attributes remain separable—are supported by measured TPR/FPR on constructed trial sets. Eq. (5) is a standard log-likelihood-ratio bound from Thudi et al. [16] and is applied to observed (FPR, TPR) operating points; it is not a parameter fitted to the paper's own definitions, nor is it used to generate the TPR/FPR values it summarizes. The DP-style Definitions 1–3 are formal target notions, not inputs that the experiments recover by construction. The self-citations—Vox-Profile [19] and the group's prior federated-learning work [11]—serve as data source, embedding extractor, and a baseline framing; they are not invoked as a uniqueness theorem or as the sole justification for the main result. No equation reduces to another by construction, and no fitted parameter is renamed as a prediction. Some reported low-FPR estimates may be statistically fragile (e.g., the small number of negative trials at FPR = 1e-4, and an internal inconsistency between the text and Table 1 for FT-Avg at L=60), but such robustness concerns are correctness risks, not circularity. The evaluation is self-contained against external benchmarks and prior MIA methodology, so no circular step is present.
Axiom & Free-Parameter Ledger
free parameters (2)
- delta = 1e-4 =
1e-4
- FPR operating points 1e-2, 1e-3, 1e-4 =
1e-2, 1e-3, 1e-4
axioms (6)
- domain assumption The low-FPR regime is the correct operationalization of speech privacy.
- standard math Eq (5), the log-likelihood-ratio bound from Thudi et al. [16], upper-bounds the relevant privacy loss.
- domain assumption The informed attacker threat model is representative of real-world adversaries.
- domain assumption Accent and gender labels from Vox-Profile and the coarse VCTK/ParaSpeechCaps grouping are reliable ground truth.
- domain assumption Same/different speaker and attribute pair trials on anonymized utterances capture re-identification and attribute inference.
- domain assumption Anonymizing each utterance four times with different speakers of desired speaker attributes supports ATTR-PRIVACY-STRICT.
invented entities (1)
-
VoxGuard benchmark
no independent evidence
read the original abstract
Voice anonymization aims to conceal speaker identity and attributes while preserving intelligibility, but current evaluations rely almost exclusively on Equal Error Rate (EER) that obscures whether adversaries can mount high-precision attacks. We argue that privacy should instead be evaluated in the low false-positive rate (FPR) regime, where even a small number of successful identifications constitutes a meaningful breach. To this end, we introduce VoxGuard, a framework grounded in differential privacy and membership inference that formalizes two complementary notions: User Privacy, preventing speaker re-identification, and Attribute Privacy, protecting sensitive traits such as gender and accent. Across synthetic and real datasets, we find that informed adversaries, especially those using fine-tuned models and max-similarity scoring, achieve orders-of-magnitude stronger attacks at low-FPR despite similar EER. For attributes, we show that simple transparent attacks recover gender and accent with near-perfect accuracy even after anonymization. Our results demonstrate that EER substantially underestimates leakage, highlighting the need for low-FPR evaluation, and recommend VoxGuard as a benchmark for evaluating privacy leakage.
Figures
Reference graph
Works this paper leans on
-
[1]
General Data Protection Regulation.https://gdpr-info.eu/, 2016
European Union EU. General Data Protection Regulation.https://gdpr-info.eu/, 2016
2016
-
[2]
V oicepm: A robust privacy measurement on voice anonymity
Shaohu Zhang, Zhouyu Li, and Anupam Das. V oicepm: A robust privacy measurement on voice anonymity. InProceedings of the 16th ACM Conference on Security and Privacy in Wireless and Mobile Networks, pages 215–226, 2023
2023
-
[3]
Speaker-dependent audio-visual emotion recognition.personal
Sanaul Haq. Speaker-dependent audio-visual emotion recognition.personal. ee. surrey. ac. uk, 2009
2009
-
[4]
Addressing challenges in speaker anonymization to maintain utility while ensuring privacy of pathological speech.Communications Medicine, 4(1):182, 2024
Soroosh Tayebi Arasteh, Tomás Arias-Vergara, Paula Andrea Pérez-Toro, Tobias Weise, Kai Packhäuser, Maria Schuster, Elmar Noeth, Andreas Maier, and Seung Hee Yang. Addressing challenges in speaker anonymization to maintain utility while ensuring privacy of pathological speech.Communications Medicine, 4(1):182, 2024
2024
-
[5]
Obfuscation via pitch-shifting for balancing privacy and diagnostic utility in voice-based cognitive assessment
Meysam Ahangaran, Nauman Dawalatabad, Cody Karjadi, James Glass, Rhoda Au, and Vijaya B Kolachalama. Obfuscation via pitch-shifting for balancing privacy and diagnostic utility in voice-based cognitive assessment. Alzheimer’s & Dementia, 21(3):e70032, 2025
2025
-
[6]
The voiceprivacy 2024 challenge evaluation plan.arXiv preprint arXiv:2404.02677, 2024
Natalia Tomashenko, Xiaoxiao Miao, Pierre Champion, Sarina Meyer, Xin Wang, Emmanuel Vincent, Michele Panariello, Nicholas Evans, Junichi Yamagishi, and Massimiliano Todisco. The voiceprivacy 2024 challenge evaluation plan.arXiv preprint arXiv:2404.02677, 2024
Pith/arXiv arXiv 2024
-
[7]
Andreas Nautsch, Jose Patino, Natalia Tomashenko, Junichi Yamagishi, Paul-Gauthier Noé, Jean-François Bonastre, Massimiliano Todisco, and Nicholas Evans. The privacy zebra: Zero evidence biometric recognition assessment.arXiv preprint arXiv:2005.09413, 2020
Pith/arXiv arXiv 2005
-
[8]
Membership inference attacks from first principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022
1914
-
[9]
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017
2017
-
[10]
Differential privacy
Cynthia Dwork. Differential privacy. InInternational colloquium on automata, languages, and programming, pages 1–12. Springer, 2006
2006
-
[11]
Tiantian Feng, Raghuveer Peri, and Shrikanth Narayanan. User-level differential privacy against attribute inference attack of speech emotion recognition in federated learning.arXiv preprint arXiv:2204.02500, 2022
Pith/arXiv arXiv 2022
-
[12]
Rui Wang, Liping Chen, Kong Aik Lee, and Zhen-Hua Ling. Asynchronous voice anonymization using adversarial perturbation on speaker embedding.arXiv preprint arXiv:2406.08200, 2024
Pith/arXiv arXiv 2024
-
[13]
Differentially private speaker anonymization.arXiv preprint arXiv:2202.11823, 2022
Ali Shahin Shamsabadi, Brij Mohan Lal Srivastava, Aurélien Bellet, Nathalie Vauquier, Emmanuel Vincent, Mohamed Maouche, Marc Tommasi, and Nicolas Papernot. Differentially private speaker anonymization.arXiv preprint arXiv:2202.11823, 2022
Pith/arXiv arXiv 2022
-
[14]
Adversarial disentanglement of speaker representation for attribute-driven privacy preservation
Paul-Gauthier Noé, Mohammad Mohammadamini, Driss Matrouf, Titouan Parcollet, Andreas Nautsch, and Jean- François Bonastre. Adversarial disentanglement of speaker representation for attribute-driven privacy preservation. arXiv preprint arXiv:2012.04454, 2020
Pith/arXiv arXiv 2012
-
[15]
Legally validated evaluation framework for voice anonymization
Nathalie Vauquier, Brij Mohan Lal Srivastava, Seyed Ahmad Hosseini, and Emmanuel Vincent. Legally validated evaluation framework for voice anonymization. InProc. Interspeech 2025, pages 3229–3233, 2025
2025
-
[16]
Bounding membership inference.arXiv preprint arXiv:2202.12232, 2022
Anvith Thudi, Ilia Shumailov, Franziska Boenisch, and Nicolas Papernot. Bounding membership inference.arXiv preprint arXiv:2202.12232, 2022
Pith/arXiv arXiv 2022
-
[17]
Xtts: a massively multilingual zero-shot text-to-speech model
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904, 2024
Pith/arXiv arXiv 2024
-
[18]
Libritts-r: A restored multi-speaker text-to-speech corpus.arXiv preprint arXiv:2305.18802, 2023
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. Libritts-r: A restored multi-speaker text-to-speech corpus.arXiv preprint arXiv:2305.18802, 2023
Pith/arXiv arXiv 2023
-
[19]
Tiantian Feng, Jihwan Lee, Anfeng Xu, Yoonjeong Lee, Thanathai Lertpetchpun, Xuan Shi, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Dani Byrd, et al. V ox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits.arXiv preprint arXiv:2505.14648, 2025
Pith/arXiv arXiv 2025
-
[20]
Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. Version 0.92, [sound], 2019. 7 PREPRINT
2019
-
[21]
Scaling rich style-prompted text-to-speech datasets.arXiv preprint arXiv:2503.04713, 2025
Anuj Diwan, Zhisheng Zheng, David Harwath, and Eunsol Choi. Scaling rich style-prompted text-to-speech datasets.arXiv preprint arXiv:2503.04713, 2025
arXiv 2025
-
[22]
Pierre Champion. Anonymizing speech: Evaluating and designing speaker anonymization techniques.arXiv preprint arXiv:2308.04455, 2023
Pith/arXiv arXiv 2023
-
[23]
Spamcop: A spam classification & organization program
Patrick Pantel, Dekang Lin, et al. Spamcop: A spam classification & organization program. InProceedings of AAAI-98 workshop on learning for text categorization, pages 95–98. Citeseer, 1998
1998
-
[24]
Better malware ground truth: Techniques for weighting anti-virus vendor labels
Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, An- thony D Joseph, and J Doug Tygar. Better malware ground truth: Techniques for weighting anti-virus vendor labels. InProceedings of the 8th ACM Workshop on Artificial Intelligence and Security, pages 45–56, 2015
2015
-
[25]
Detecting credential spearphishing in enterprise settings
Grant Ho, Aashish Sharma, Mobin Javed, Vern Paxson, and David Wagner. Detecting credential spearphishing in enterprise settings. In26th USENIX security symposium (USENIX security 17), pages 469–485, 2017
2017
-
[26]
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143, 2020
Pith/arXiv arXiv 2005
-
[27]
Vauquier and et al
R. Vauquier and et al. Beyond eer: Linkability and singling-out for voice. InProc. Interspeech, 2025. 8
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.