EASY: Emotion-aware Speaker Anonymization via Factorized Distillation

Eng Siong Chng; Hexin Liu; Jixun Yao; Lei Xie

REVIEW 5 major objections 5 minor 39 references

EASY: Emotion-aware Speaker Anonymization via Factorized Distillation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EASY claims to be the first emotion-aware speaker anonymization framework that sequentially factorizes identity, content, and emotion into separate subspaces, and reports superior privacy and emotion preservation over all VoicePrivacy…

desk verdict Incremental but credible emotion-aware speaker anonymization system; the emotion UAR evaluation needs backend disclosure before the headline claim can be trusted. read the letter →

arxiv 2505.15004 v2 pith:WFHO27MY submitted 2025-05-21 eess.AS cs.SD

Jixun Yao , Hexin Liu , Eng Siong Chng , Lei Xie This is my paper

classification eess.AScs.SD

keywords speakeranonymizationvoiceprivacyfactorizeddistillationsequentialdisentanglementresidualvectorquantizationemotionpreservationemotion2vecWavLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EASY is an attempt to get speaker anonymization to preserve emotion, not just words and intelligibility. Most voice-conversion anonymizers pull speech apart into content and speaker in parallel and let emotion fall where it may; EASY instead separates three attributes in a fixed order, stripping the speaker first, then letting a cascade of residual quantizers keep the words in one layer and the emotion in the rest. If the claim holds, spoken-dialogue services could anonymize audio before sending it to the cloud and still receive the user's emotional state, which is the paper's stated motivation. On the VoicePrivacy 2024 benchmark the paper reports the best privacy and emotion scores among every system compared, with word error only slightly above original audio and below every official baseline; the one rival with marginally lower word error scores far worse on both privacy and emotion.

What carries the argument

The central object is the residual vector quantizer used as a serial disentangler: a stack of eight quantizer layers in which each layer only encodes what the previous layers failed to represent, so attributes separate by position in the stack rather than by a joint objective. In EASY the first quantizer is locked to words by a cross-entropy loss against k-means-quantized WavLM features, while the remaining quantizers are locked to emotion by KL divergence against emotion2vec, with gradient-reversal classifiers suppressing speaker and semantic labels from the residual. The speaker branch does the same job one step earlier: a classifier-aligned utterance embedding is subtracted from the frame-level representation to yield the speaker-independent input to the stack. The anonymized identity at inference is a convex blend $s_{\mathrm{anon}} = \alpha\bar{s} + (1-\alpha)\hat{s}$ of an averaged pseudo-speaker vector and a sample from a Gaussian.

What would settle it

Re-run the VoicePrivacy 2024 privacy evaluation exactly, but with the speaker-verification model fine-tuned on EASY-anonymized speech from the same identities; if equal error rate drops well below the reported ~46%, the residual quantizer layers leaked identity that the standard attacker did not detect. A simpler probe that the paper itself partially runs: reconstruct audio from only the residual quantizer layers (Table 3 finds WER 26.54% for that reconstruction) and run speaker verification against the original identities on it, since recovering a substantial fraction of identities would falsify the claim that the residual contains minimal speaker identity.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that identity, words, and emotion can be separated in sequence rather than in parallel. A supervised classifier collapses the utterance into a speaker vector that is subtracted from the frame-level representation; the first layer of a residual vector-quantization stack is then pinned to linguistic content through distillation against k-means-tokenized WavLM features; and the remaining quantizer layers are aligned to emotion2vec while gradient-reversal classifiers try to push residual speaker and content information out of them. At inference the encoders are dropped and only the speaker vector is replaced, by a weighted blend of an averaged pseudo-speaker and a Gaussian draw, leaving the content and emotion codes untouched. The result, the paper argues, is that privacy, intelligibility, and emotional state are each carried by a separate subspace, so constraining each independently prevents the information leakage that parallel disentanglement suffers.

Load-bearing premise

Everything hinges on the assumption that the three information streams genuinely separate in sequence: after the speaker vector is subtracted and the first quantizer is pinned to words, what remains is taken to be almost pure emotion with only negligible traces of who spoke or what was said; if that leftover still hides identity or content, the privacy or the clarity will quietly degrade.

Editorial extensions

If this is right

Privacy and emotional expressiveness need not trade off: EASY reports the best of both on the VPC 2024 benchmark, so spoken-dialogue services could anonymize audio before cloud processing and still receive the user's emotional state.
The sequential design makes each attribute addressable: swapping the speaker vector alone changes identity, while the quantizer split means content and emotion could be edited independently in future systems.
Because emotion is tied to the residual quantizer layers, a system that wants to modulate emotional intensity could adjust those layers without retraining the whole model.
Automatic evaluation of anonymization can now include a third column, emotion recall, alongside privacy and intelligibility, which the paper argues should become a standard part of VoicePrivacy-style benchmarks.
The ablation results attribute a specific role to each loss: removing identity distillation sharply drops both privacy and emotion scores, removing semantic distillation damages intelligibility, and removing emotion distillation lowers both privacy and emotion, supporting the claim that each factorized constraint is necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The residual-factorization premise is the part to probe: the paper's evidence that the later quantizers contain 'minimal speaker identity and linguistic content' is behavioral ablations, not a direct measurement; a verification model trained on anonymized data rather than original speech would test whether identity truly stayed out of the residual.
Because emotion2vec and the IEMOCAP evaluation are anchored in acted emotional speech, the strong UAR numbers may not carry over to spontaneous conversation; evaluation on naturally emotional dialogue would be a harder test.
The same serial residual-VQ scheme could be repurposed for other attribute pairs, such as accent, age, or speaking style, since the separation mechanism itself is agnostic to which teachers are chosen for each layer.
A stronger privacy claim would come from pushing the residual layers' identity leakage toward zero: adversarially retraining the speaker encoder against a stronger attacker could move EER closer to the 50% chance ceiling rather than the reported ~46%.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

EASY is a solid incremental system paper, not a conceptual breakthrough. What's new: it extends the authors' MUSA serial-RVQ disentanglement line by adding a third factor—emotion—distilled from emotion2vec, and shows competitive VPC 2024 numbers. The architecture is clearly described, the ablations in Tables 2 and 3 line up with the design, and the privacy gains (EER near 46%) are the strongest in the table. That part deserves credit.

The soft spots are real but mostly fixable. First, the emotion-preservation claim rests entirely on UAR, and the paper never says which emotion-recognition backend produces that UAR. Since emotion2vec is the distillation teacher in Eq. (3), any overlap between teacher and evaluator would give EASY an unfair advantage over baselines; the margins (64.19 vs 62.93) are small enough that a shared backend could explain them. This needs to be disclosed, or the evaluation rerun with an independent backend. Second, all numbers are single-run point estimates; no error bars, no significance tests, so the "outperforms all baselines" claim is weaker than the prose suggests. Third, the abstract says EASY outperforms all baselines, but OH actually has lower WER (2.36/2.48 vs 2.71/2.69); the body acknowledges this, so the abstract overclaims. Fourth, the residual-factorization premise—that after subtracting the speaker vector, VQ-1 carries content and VQ-2:8 carries emotion—is supported only by indirect ablation, and the w/o s row in Table 3 hints that the speaker encoder still captures some non-identity information. That is a modeling assumption worth probing, not a fatal flaw.

Novelty relative to the authors' own MUSA system is not fully isolated; the paper could state more explicitly what MUSA lacks and what EASY adds beyond the emotion teacher. Still, the WavLM + emotion2vec factorized distillation is a legitimate configuration not in the prior literature.

Bottom line: worth a serious referee. The evaluation-disclosure issue is the load-bearing concern and can be fixed in a paragraph. If the authors release code or at least specify the UAR backend, the empirical claims become checkable. I'd bring it to a reading group focused on voice privacy, and I'd cite it if I worked in that area.

Referee Report

5 major / 5 minor

Summary. The paper proposes EASY, an autoencoder-style speaker anonymization system that sequentially factorizes speech into speaker identity, linguistic content, and emotion. A speaker encoder is supervised with speaker-ID classification; the first RVQ quantizer is distilled toward WavLM semantic tokens; the residual quantizer is distilled toward emotion2vec embeddings with gradient-reversal adversarial losses; anonymization mixes a pooled pseudo-speaker with a Gaussian-sampled speaker vector. Evaluation on VPC2024 and IEMOCAP reports EER, WER, and UAR, and the paper claims state-of-the-art privacy and emotion preservation, with ablations for each loss component.

Significance. If substantiated, EASY would be a meaningful advance: it addresses an underexplored utility dimension (emotion) in voice anonymization, uses a clean sequential RVQ formulation, and reports improvements over the main VPC2024 baselines on standard data. The component ablations and layer-reconstruction analysis are useful evidence for the disentanglement story. However, the verification currently rests on undisclosed emotion-evaluation details, single-run metrics, and an asserted residual-factorization premise; these issues must be resolved before the headline claim is accepted.

major comments (5)

[Section 3.2 / Eq. (3)] The UAR evaluation backend is never specified. Because Eq. (3) trains the residual quantizer to match emotion2vec, if the IEMOCAP UAR scorer is emotion2vec-based (or uses emotion2vec features), the emotion-preservation comparison in Table 1 is biased in favor of EASY relative to the baselines, none of which are trained against that teacher. The reported emotion margins are small (64.19 vs 62.93 on dev; 63.43 vs 60.87 on test), so this is not a negligible effect. Please disclose the exact SER model/feature backend used for UAR, or re-evaluate with an independent emotion-recognition system.
[Section 4, Tables 1-2] All metrics are point estimates without error bars, confidence intervals, or significance tests. The anonymization strategy in Eq. (7) involves random selection and Gaussian sampling, so the system is stochastic; the observed differences (e.g., EER 46.67 vs 42.45, UAR 64.19 vs 62.93) need repeated-run variance or significance testing before 'outperforms all baseline systems' is supportable. Please report multiple seeds or bootstrap intervals for at least the main comparison.
[Section 2.2, Eq. (3)] The text says 'the remaining quantizers' are emotion-distilled, but Eq. (3) defines the KL and adversarial losses only for q_n. If the emotion loss applies to q_2,...,q_n, the equation should contain a sum over i; if it applies only to the last quantizer, the roles of the intermediate residual layers should be explained. This ambiguity directly affects the sequential-disentanglement claim and the interpretation of Table 3's VQ-2:8 group.
[Section 2.2, residual factorization premise] The statement that after VQ-1 'the residual information contains minimal speaker identity and linguistic content' is asserted rather than measured. Table 3 shows VQ-2:8 reconstruction has WER 26.54% and UAR 41.28%, which are clearly degraded but far from chance, and the table is an indirect reconstruction probe rather than a leakage test on the quantizer embeddings themselves. Please add direct probing (speaker-ID, emotion, and ASR classification accuracy on q_n embeddings) or otherwise quantify residual leakage, since the privacy and emotion-separation claims depend on this premise.
[Abstract and Section 4.1] The sentence 'outperforms all baseline systems' in the Abstract is too strong: Table 1 shows that T9 and OH achieve lower WER (2.33/2.37 and 2.36/2.48) than EASY (2.71/2.69). The more precise statement in Section 4.1 that EASY outperforms baselines in privacy protection and emotion preservation should replace the unqualified claim, and the WER trade-off should be acknowledged explicitly.

minor comments (5)

[Table 1 caption] The caption does not state that the '+/-' entries are differences relative to B1; please make this explicit in the caption or table notes.
[Figure 1 and abstract] There are typos: 'Anoniymized' in Figure 1 and 'V oicePrivacy' in the Abstract; please proofread the manuscript.
[Eq. (6)] Equation (6) writes lambda_r as 'hyper-parameters'; please clarify whether all other loss terms have weight 1 and whether lambda_r was fixed at 45 from prior work or tuned on the development set.
[Section 2.2 and Eq. (3)] The notation q_n appears before n is defined; Section 3.1 states that 8 quantizers are used, so please define N and use consistent indexing q_1,...,q_N throughout.
[Section 3.2] The paper says metrics follow VPC 2024, but the exact ASV, ASR, and SER model versions used to produce EER, WER, and UAR are not listed; please provide model identifiers or cite the specific VPC evaluation-plan section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EASY is evaluated on external VPC 2024 benchmarks; the emotion2vec teacher concern is conditional and not established by the paper.

full rationale

The paper's central claims are empirical, benchmarked against the external VoicePrivacy 2024 evaluation (EER, WER, UAR following the VPC 2024 configuration). The training losses (Eqs. 1-6) are standard distillation and quantization objectives with external teachers (WavLM, emotion2vec); the evaluation metrics are not defined by the paper itself. The only potentially circular configuration would be if the VPC 2024 UAR backend used emotion2vec features, since Eq. (3) trains the residual quantizer against emotion2vec. However, the paper merely states that UAR follows VPC 2024 settings, and no information in the paper establishes that the VPC backend uses emotion2vec; without that, the concern is conditional, not a demonstrated reduction. Self-citations to the authors' prior VPC submissions (e.g., [19], [20]) are normal and not load-bearing: the baseline T10 is an independently published system used for comparison, and the novel architecture does not depend on any self-cited theorem or result. Therefore no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five domain assumptions about residual factorization, speaker subtraction, teacher fidelity, metric validity, and GRL leakage removal, plus four hand-chosen hyperparameters. No new physical or architectural entities are invented; the speaker vector pool is a data structure rather than a postulated entity.

free parameters (4)

alpha (anonymization interpolation weight) = not reported
Equation 7 blends averaged and randomly sampled speaker identities; this sets the privacy/utility tradeoff, but the paper gives no chosen value or tuning procedure.
Number of k-means clusters for WavLM semantic tokens = not reported
Equation 2 uses k-means tokens as linguistic distillation labels; the cluster count determines token granularity, yet it is not stated.
Number of RVQ layers N = 8
Section 3.1 fixes eight residual quantizers; the split between linguistic and emotional layers depends on this hand-chosen capacity.
Reconstruction loss weight lambda_r = 45
Equation 6 sets lambda_r=45 following [27]; it balances reconstruction against distillation and adversarial terms.

assumptions (5)

domain assumption Residual RVQ decomposition factorizes attributes by layer order: the first layer carries linguistic content and later layers carry emotion.
Section 2.2 states 'we choose the first VQ layer to represent linguistic content' and 'the residual information contains minimal speaker identity and linguistic content'; the entire emotion branch depends on this.
domain assumption Subtracting a supervised speaker vector from frame-level features yields a speaker-independent representation.
Section 2.2 Speaker distillation assumes the speaker classifier captures identity without removing content or emotion.
domain assumption WavLM 6th-layer k-means tokens and emotion2vec embeddings are faithful teachers for linguistic content and emotion.
Section 2.2 uses them as distillation targets; if a teacher encodes the wrong attributes, the factorization is mis-specified.
domain assumption VPC2024 EER, WER, and UAR are valid proxies for privacy, intelligibility, and emotion preservation.
Section 3.2 inherits these metrics from the challenge without questioning their validity or the statistical reliability of single-run scores.
domain assumption Gradient-reversal adversarial classifiers make residual quantizers invariant to speaker identity and semantic tokens.
Equation 3 adds speaker- and semantic-GRL losses; the method assumes GRL training actually removes leakage from the emotion representation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EASY: Emotion-aware Speaker Anonymization via Factorized Distillation." pith.science (2026). https://pith.science/paper/WFHO27MY

@misc{pith2026250515004,
  author       = {Pith},
  title        = {Pith review of: EASY: Emotion-aware Speaker Anonymization via Factorized Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFHO27MY}},
  note         = {Machine review of arXiv:2505.15004}
}

read the original abstract

Emotion plays a significant role in speech interaction, conveyed through tone, pitch, and rhythm, enabling the expression of feelings and intentions beyond words to create a more personalized experience. However, most existing speaker anonymization systems employ parallel disentanglement methods, which only separate speech into linguistic content and speaker identity, often neglecting the preservation of the original emotional state. In this study, we introduce EASY, an emotion-aware speaker anonymization framework. EASY employs a novel sequential disentanglement process to disentangle speaker identity, linguistic content, and emotional representation, modeling each speech attribute in distinct subspaces through a factorized distillation approach. By independently constraining speaker identity and emotional representation, EASY minimizes information leakage, enhancing privacy protection while preserving original linguistic content and emotional state. Experimental results on the VoicePrivacy Challenge official datasets demonstrate that our proposed approach outperforms all baseline systems, effectively protecting speaker privacy while maintaining linguistic content and emotional state.

Figures

Figures reproduced from arXiv: 2505.15004 by the authors.

**Figure 1.** The overall architecture of our proposed speaker anonymization framework. 2. EASY 2.1. Overview Our proposed EASY framework employs an auto-encoder architecture comprising four individual encoders, a decoder, and a residual bottleneck module with multiple RVQ blocks. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

[1]

Introduction With the advent of large language models, recent advance- ments in spoken dialogue systems, such as GPT-4o [1], have demonstrated remarkable capabilities in human-computer in- teraction [2, 3]. These models can perceive subtle emotional changes in speech and provide context-aware responses, sig- nificantly enhancing usability and interactivit...

arXiv 2024
[2]

EASY 2.1. Overview Our proposed EASY framework employs an auto-encoder ar- chitecture comprising four individual encoders, a decoder, and a residual bottleneck module with multiple RVQ blocks. As illus- trated in Figure 1, the speech encoder Ec and speaker encoder Ep are first employed to extract frame-level and utterance-level representations from the sp...

work page
[3]

Dataset We train EASY using the training portions of both the Lib- riSpeech [28] and LibriTTS [29] corpora

Experimental Setup 3.1. Dataset We train EASY using the training portions of both the Lib- riSpeech [28] and LibriTTS [29] corpora. For evaluation, we follow the VPC 2024 configuration [7], using LibriSpeech-dev- clean and LibriSpeech-test-clean for privacy and utility evalua- tion. Additionally, the IEMOCAP [30] development and evalu- ation sets are used...

work page 2024
[4]

Experimental Results 4.1. Privacy and Utility Evaluation We evaluate EASY using the VPC evaluation datasets and com- pare its performance with the official VPC baseline systems and state-of-the-art anonymization systems. As shown in Table 1, the EER results for B1 and B2 are only slightly higher than those of the original speech, indicating poor privacy p...

work page
[5]

Speaker identity is first disentangled using a supervised classifier, producing a speaker-independent representation

Conclusion In this study, we present EASY , an emotion-aware speaker anonymization framework that sequentially disentangles speaker identity, linguistic content, and emotional represen- tation through a factorized distillation process, explicitly modeling each speech attribute in distinct subspaces. Speaker identity is first disentangled using a supervise...

work page 2024
[6]

Acknowledgement This work was supported by Tencent and Tencent-NTU Joint Research Laboratory (CENTURY), Nanyang Technological University, Singapore

work page
[7]

OpenAI, “GPT-4o.” https://openai.com/index/hello-gpt-4o/, 2024

work page 2024
[8]

Moshi: a speech-text foundation model for real-time dialogue,

A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

arXiv 2024

Show all 39 references

[9]

Mini-omni: Language models can hear, talk while thinking in streaming,

Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,” arXiv preprint arXiv:2408.16725 , 2024

2024 arXiv
[10]

Adapting general disentanglement-based speaker anonymization for enhanced emotion preservation,

X. Miao, Y . Zhang, X. Wang, N. Tomashenko, D. C. L. Soh, and I. Mcloughlin, “Adapting general disentanglement-based speaker anonymization for enhanced emotion preservation,”arXiv preprint arXiv:2408.05928, 2024

2024 arXiv
[11]

The voiceprivacy 2020 challenge: Results and findings,

N. Tomashenko, X. Wang, E. Vincent, J. Patino, B. M. L. Srivas- tava, P.-G. No´e, A. Nautsch, N. Evans, J. Yamagishi, B. O’Brien et al., “The voiceprivacy 2020 challenge: Results and findings,” Computer Speech & Language, vol. 74, p. 101362, 2022

2020
[12]

The voiceprivacy 2022 challenge: Progress and perspec- tives in voice anonymisation,

M. Panariello, N. Tomashenko, X. Wang, X. Miao, P. Champion, H. Nourtel, M. Todisco, N. Evans, E. Vincent, and J. Yamag- ishi, “The voiceprivacy 2022 challenge: Progress and perspec- tives in voice anonymisation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

2022
[13]

The voiceprivacy 2024 challenge evaluation plan,

N. Tomashenko, X. Miao, P. Champion, S. Meyer, X. Wang, E. Vincent, M. Panariello, N. Evans, J. Yamagishi, and M. Todisco, “The voiceprivacy 2024 challenge evaluation plan,” arXiv preprint arXiv:2404.02677, 2024

2024 arXiv
[14]

Speaker anonymisation using the mcadams co- efficient,

J. Patino, N. A. Tomashenko, M. Todisco, A. Nautsch, and N. W. D. Evans, “Speaker anonymisation using the mcadams co- efficient,” in Proc. INTERSPEECH, 2021, pp. 1099–1103

2021
[15]

Improving speaker de-identification with functional data analysis of f0 trajectories,

L. Tavi, T. Kinnunen, and R. G. Hautam ¨aki, “Improving speaker de-identification with functional data analysis of f0 trajectories,” Speech Communication, vol. 140, pp. 1–10, 2022

2022
[16]

Speaker anonymiza- tion by pitch shifting based on time-scale modification,

C. O. Mawalim, S. Okada, and M. Unoki, “Speaker anonymiza- tion by pitch shifting based on time-scale modification,” in Proc. SPSC, 2022, pp. 35–42

2022
[17]

Privacy and utility of x-vector based speaker anonymiza- tion,

B. M. L. Srivastava, M. Maouche, M. Sahidullah, E. Vincent, A. Bellet, M. Tommasi, N. Tomashenko, X. Wang, and J. Yam- agishi, “Privacy and utility of x-vector based speaker anonymiza- tion,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2383–2395, 2022

2022
[18]

Language-independent speaker anonymization using orthogonal householder neural network,

X. Miao, X. Wang, E. Cooper, J. Yamagishi, and N. Tomashenko, “Language-independent speaker anonymization using orthogonal householder neural network,” arXiv preprint arXiv:2305.18823 , 2023

2023 arXiv
[19]

Dis- tinguishable speaker anonymization based on formant and fun- damental frequency scaling,

J. Yao, Q. Wang, Y . Lei, P. Guo, L. Xie, N. Wang, and J. Liu, “Dis- tinguishable speaker anonymization based on formant and fun- damental frequency scaling,” in ICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

2023
[20]

Prosody is not identity: A speaker anonymization approach using prosody cloning,

S. Meyer, F. Lux, J. Koch, P. Denisov, P. Tilli, and N. T. Vu, “Prosody is not identity: A speaker anonymization approach using prosody cloning,” inICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 111–115

2023
[21]

Distinctive and natural speaker anonymization via singular value transformation- assisted matrix,

J. Yao, Q. Wang, P. Guo, Z. Ning, and L. Xie, “Distinctive and natural speaker anonymization via singular value transformation- assisted matrix,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2944–2956, 2024

2024
[22]

Speaker anonymization using neural audio codec language models,

M. Panariello, F. Nespoli, M. Todisco, and N. Evans, “Speaker anonymization using neural audio codec language models,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 4725–4729

2024
[23]

MUSA: Multi-lingual speaker anonymization via serial disen- tanglement,

J. Yao, Q. Wang, P. Guo, Z. Ning, Y . Yang, Y . Pan, and L. Xie, “MUSA: Multi-lingual speaker anonymization via serial disen- tanglement,” IEEE Transactions on Audio, Speech and Language Processing, 2025

2025
[24]

Salt: Dis- tinguishable speaker anonymization through latent space transfor- mation,

Y . Lv, J. Yao, P. Chen, H. Zhou, H. Lu, and L. Xie, “Salt: Dis- tinguishable speaker anonymization through latent space transfor- mation,” in2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), 2023, pp. 1–8

2023
[25]

NWPU- ASLP system for the voiceprivacy 2022 challenge,

J. Yao, Q. Wang, L. Zhang, P. Guo, Y . Liang, and L. Xie, “NWPU- ASLP system for the voiceprivacy 2022 challenge,”arXiv preprint arXiv:2209.11969, 2022

2022 arXiv
[26]

NPU-NTU system for voice privacy 2024 challenge,

J. Yao, N. Kuzmin, Q. Wang, P. Guo, Z. Ning, D. Guo, K. A. Lee, E.-S. Chng, and L. Xie, “NPU-NTU system for voice privacy 2024 challenge,” arXiv preprint arXiv:2409.04173, 2024

2024 arXiv
[27]

WavLM: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

2022
[28]

Emotion2vec: Self-supervised pre-training for speech emotion representation,

Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “Emotion2vec: Self-supervised pre-training for speech emotion representation,” arXiv preprint arXiv:2312.15185, 2023

2023 arXiv
[29]

Language modeling with gated convolutional networks,

Y . N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proc. ICML , vol. 70, 2017, pp. 933–941

2017
[30]

Neural discrete representa- tion learning,

A. Van Den Oord, O. Vinyals et al., “Neural discrete representa- tion learning,” Proc. NeurIPS, vol. 30, 2017

2017
[31]

Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,” arXiv preprint arXiv:2308.16692, 2023

2023 arXiv
[32]

HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” inProc. NeurIPS, 2020

2020
[33]

Hifi- codec: Group-residual vector quantization for high fidelity audio codec,

D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “Hifi- codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765, 2023

2023 arXiv
[34]

Lib- riSpeech: an asr corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- riSpeech: an asr corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206–5210

2015
[35]

LibriTTS: A corpus derived from librispeech for text- to-speech,

H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from librispeech for text- to-speech,” arXiv preprint arXiv:1904.02882, 2019

1904 arXiv
[36]

IEMOCAP: Interactive emotional dyadic motion capture database,

C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

2008
[37]

Anonymizing speech: Evaluating and de- signing speaker anonymization techniques,

P. Champion, “Anonymizing speech: Evaluating and de- signing speaker anonymization techniques,” arXiv preprint arXiv:2308.04455, 2023

2023 arXiv
[38]

Hltcoe jhu sub- mission to the voice privacy challenge 2024,

H. L. Xinyuan, Z. Cai, A. Garg, K. Duh, L. P. Garc ´ıa-Perera, S. Khudanpur, N. Andrews, and M. Wiesner, “Hltcoe jhu sub- mission to the voice privacy challenge 2024,” arXiv preprint arXiv:2409.08913, 2024

2024 arXiv
[39]

System descrip- tion: Speaker anonymization system with sentiment transfer and feature interpolation,

T. Tan, S. Liu, Y . Duan, S. Zhao, and X. Shao, “System descrip- tion: Speaker anonymization system with sentiment transfer and feature interpolation,” voiceprivacychallenge.org, 2024

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Introduction With the advent of large language models, recent advance- ments in spoken dialogue systems, such as GPT-4o [1], have demonstrated remarkable capabilities in human-computer in- teraction [2, 3]. These models can perceive subtle emotional changes in speech and provide context-aware responses, sig- nificantly enhancing usability and interactivit...

arXiv 2024

[2] [2]

EASY 2.1. Overview Our proposed EASY framework employs an auto-encoder ar- chitecture comprising four individual encoders, a decoder, and a residual bottleneck module with multiple RVQ blocks. As illus- trated in Figure 1, the speech encoder Ec and speaker encoder Ep are first employed to extract frame-level and utterance-level representations from the sp...

work page

[3] [3]

Dataset We train EASY using the training portions of both the Lib- riSpeech [28] and LibriTTS [29] corpora

Experimental Setup 3.1. Dataset We train EASY using the training portions of both the Lib- riSpeech [28] and LibriTTS [29] corpora. For evaluation, we follow the VPC 2024 configuration [7], using LibriSpeech-dev- clean and LibriSpeech-test-clean for privacy and utility evalua- tion. Additionally, the IEMOCAP [30] development and evalu- ation sets are used...

work page 2024

[4] [4]

Experimental Results 4.1. Privacy and Utility Evaluation We evaluate EASY using the VPC evaluation datasets and com- pare its performance with the official VPC baseline systems and state-of-the-art anonymization systems. As shown in Table 1, the EER results for B1 and B2 are only slightly higher than those of the original speech, indicating poor privacy p...

work page

[5] [5]

Speaker identity is first disentangled using a supervised classifier, producing a speaker-independent representation

Conclusion In this study, we present EASY , an emotion-aware speaker anonymization framework that sequentially disentangles speaker identity, linguistic content, and emotional represen- tation through a factorized distillation process, explicitly modeling each speech attribute in distinct subspaces. Speaker identity is first disentangled using a supervise...

work page 2024

[6] [6]

Acknowledgement This work was supported by Tencent and Tencent-NTU Joint Research Laboratory (CENTURY), Nanyang Technological University, Singapore

work page

[7] [7]

OpenAI, “GPT-4o.” https://openai.com/index/hello-gpt-4o/, 2024

work page 2024

[8] [8]

Moshi: a speech-text foundation model for real-time dialogue,

A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

arXiv 2024