Pith. sign in

REVIEW 3 major objections 9 minor 49 references

SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment

T0 review · 3 major / 9 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SemAlignVC claims that aligning an audio semantic encoder to BERT text embeddings with monotonic alignment strips speaker timbre from the representation, enabling zero-shot voice conversion without explicit speaker embeddings.

desk verdict Genuinely new alignment objective and a striking speaker-classification result, but the abstract overclaims intelligibility and the BERT-target assumption needs ablated evidence. read the letter →

arxiv 2507.09070 v1 pith:QUP4IQW4 submitted 2025-07-11 eess.AS cs.SD

Shivam Mehta , Yingru Liu , Zhenyu Tang , Kainan Peng , Vimal Manohar , Shun Zhang , Mike Seltzer , Qing He
show 1 more author
Mingbo Ma
This is my paper
classification eess.AScs.SD
keywords zero-shotvoiceconversiontimbreleakagesemanticalignmentrepresentationdisentanglementtext-audiospeakerprivacyflowmatchingBERTembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SemAlignVC tries to remove timbre leakage from zero-shot voice conversion: when speech is converted to a new speaker's voice, traces of the original speaker should not survive. The paper's central move is to align the audio representation to a text-only semantic representation (BERT) using Monotonic Alignment Search, so any speaker-specific acoustic information has nowhere to hide. The paper reports that the learned content representation identifies the source speaker only 2.84% of the time, near chance, and that the converted speech beats three baselines on listener-rated timbre similarity and naturalness, with competitive intelligibility. If correct, this means a voice converter can work without storing or using an explicit speaker embedding, which also makes the pipeline more privacy-preserving.

What carries the argument

The central mechanism is the SemAlign objective. The semantic encoder $Q_\phi$ produces a frame-level audio representation $\hat{a}_s$; the transcript is embedded by a pretrained text-only transformer (BERT) into $\tau_s$, and Monotonic Alignment Search (MAS) finds a monotonic path that upsamples $\tau_s$ to the audio length. Minimizing the mean-squared error between the upsampled $\tau_s$ and $\hat{a}_s$ forces $Q_\phi$ to encode only what the text already contains. A CTC loss preserves phonetic detail, and gradient flow from the autoregressive decoder into $Q_\phi$ is blocked so speaker cues cannot be written back.

What would settle it

Take a held-out set of unseen source-target speaker pairs, run SemAlignVC conversions, and compute cosine similarity between the converted audio and both the source and target speaker embeddings. If source-to-converted similarity is not at chance level while target-to-converted similarity is high, timbre leakage persists; alternatively, an ablation that removes the SemAlign loss should make speaker-classification accuracy on $Q_\phi$ jump far above 2.84% if the alignment is the active mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that speaker timbre can be filtered out of a neural codec representation by forcing it to line up with frozen text embeddings, instead of by adding a bottleneck or a speaker-verification embedding. The semantic encoder $Q_\phi$ is trained so its outputs match a BERT embedding of the transcript, upsampled to audio length through Monotonic Alignment Search, while a CTC loss keeps the words intact. The result is a content representation that is nearly speaker-independent: a classifier reaches 2.84% accuracy on it, compared with 71.7% for HuBERT tokens, 96.7% for EnCodec tokens, and 82.05% for the authors' own tokenizer. Converted speech conditioned only on this representation, normalized pitch and energy, and the target speaker's reference mel achieves the highest speaker-similarity scores among the compared systems (WavLM 0.95, ECAPA 0.82, Resemblyzer 0.89).

Load-bearing premise

The load-bearing premise is that BERT text embeddings are a speaker-independent, semantically complete target for speech; if those embeddings carry stylistic cues or lack phonetic detail, forcing audio to align with them will either leak timbre or corrupt content.

Editorial extensions

If this is right

  • Because SemAlignVC needs no speaker embedding, the converted voice is fully determined by the reference mel; this makes any-to-any conversion possible for speakers never seen in training.
  • The near-chance speaker classification on $Q_\phi$ (2.84%) means the representation can be released or stored without giving away who the original speaker was, a direct privacy benefit.
  • The text-alignment objective is tokenizer-agnostic: the paper states the audio tokenizer can be swapped, so the same disentanglement recipe applies to other neural codecs.
  • The reported WER of 12.31% shows that intelligibility survives the alignment, although it is not the best in the comparison, so the remaining weakness is fine-grained linguistic detail, not timbre.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's English-only experiments, the same alignment idea could be applied to multilingual text embeddings or phoneme sequences; if the target representation is language-neutral, the pipeline would gain cross-lingual conversion without retraining the alignment mechanism.
  • The paper's own observation of synonym substitutions in a pure TTS model suggests that text-embedding quality, not alignment, is the next bottleneck; an extension would test whether stronger text encoders lower WER below 12.31%.
  • The near-zero speaker classification accuracy does not by itself prove that style, emotion, or accent are absent; measuring those separately would tell whether SemAlign removes speaker identity broadly or only timbre.
  • A practical extension is to check whether the same representation improves speaker anonymization in downstream tasks like automatic speech recognition, where 2.84% speaker classification accuracy would make speaker linking harder.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. SemAlignVC proposes a zero-shot voice conversion pipeline that removes source timbre by training a semantic encoder Qphi with two auxiliary losses: CTC and SemAlign, where Qphi representations are monotonically aligned to BERT text embeddings via monotonic alignment search (MAS). An autoregressive transformer predicts codec tokens from Qphi, normalized pitch/energy, and a reference mel-spectrogram segment; a flow-matching acoustic model and BigVGAN synthesize the final waveform. The paper reports speaker classification accuracy of 2.84% for Qphi, the highest speaker similarity scores among KNNVC, HierSpeech++, and UniAudio on WavLM/ECAPA/Resemblyzer, the highest SMOS (3.29), and the highest DNSMOS OVRL (3.38), but a WER of 12.31% versus 8.24% for HierSpeech++ and 9.98% for UniAudio on the same 50-utterance set.

Significance. If validated, SemAlignVC is an interesting approach to speaker-information removal in codec-based voice conversion that avoids a learned speaker embedding at inference. The evaluation is comparatively careful: baselines are taken from official repositories with pretrained checkpoints, the subjective test uses longer utterances and 120 participants, and the speaker-similarity gains are consistent across three different embedding models. However, the headline intelligibility claim is contradicted by the paper's own numbers, and the main disentanglement evidence is partly a consequence of the training target rather than an independent confirmation. The core idea, aligning audio semantics to text embeddings, is worth pursuing, but the claims need substantial recalibration before the paper can be accepted.

major comments (3)
  1. [Abstract; Table 3; Section 5] The abstract states that SemAlignVC "outperforms baselines in speaker timbre similarity, intelligibility, and naturalness," but Table 3 reports WER 12.31% for SemAlignVC versus 8.24% for HierSpeech++ and 9.98% for UniAudio on the same 50-utterance set. Section 5 acknowledges that the WER is "slightly higher than HierSpeech++" and that there is "room for improvement in preserving fine-grained linguistic details," and the conclusion retreats to "comparable intelligibility." A 49% relative WER increase over HierSpeech++ is not "comparable" in the usual sense, so the abstract is internally inconsistent with the reported results. Since intelligibility is one of the three headline claims, the authors must either correct the claim or provide additional evidence (for example, statistical significance testing or articulation-level metrics) to support it. The naturalness claim is also not fully established: the DNSMOS OVRL advantage (3.38 versus 3.34) is small and reported without confidence intervals or significance tests.
  2. [Table 1; Section 3.2; Section 5] The 2.84% speaker-classification accuracy for Qphi is presented as evidence of "superior timbre removal," but Qphi is trained with an MSE loss to match BERT text embeddings (Section 3.2). BERT text embeddings are text-only and therefore contain no speaker identity by construction, and the classification head is a single layer, so the probe primarily measures linear decodability. The low accuracy is thus a consequence of the training objective rather than an independent confirmation of disentanglement. The paper should temper this interpretation and provide controls, for example the same probe on a Qphi variant trained only with CTC, nonlinear probes, or a content-retention metric such as phone accuracy, to show that the retained representation is still sufficient for linguistic content.
  3. [Section 3.2; Section 5] The BERT alignment target may be semantically but not phonetically complete. Section 5 reports that a TTS model with BERT-based representations "observed occasional word substitutions with synonyms during generation" and links this to "minor mispronunciations." This is exactly the failure mode expected if subword text embeddings collapse phonetically confusable pairs. Because SemAlign forces Qphi into the BERT embedding space, the intelligibility degradation in Table 3 is plausibly structural rather than incidental. The paper should either compare against a phonetically richer target (for example, phoneme-level embeddings) or explicitly position the intelligibility loss as a limitation of the alignment target; the current presentation leaves the load-bearing assumption — that BERT embeddings are a semantically complete and phonetically sufficient target — unsupported.
minor comments (9)
  1. [Section 3.1] The word "geneative" should be "generative."
  2. [Table 1] There is a stray percent sign in the HuBERT row ("71.7%"), and "Ourstok" should be written as "Ours (token)" for clarity.
  3. [Section 4.1] The speaker-classification experiment does not state whether the train and test speakers are disjoint; this should be clarified, as overlap would inflate all accuracies and affect the comparison.
  4. [Section 3.2; Section 4.2] The total loss for the semantic encoder is described only verbally; the authors should specify the weighting of Lsem and the CTC loss, since these are free parameters of the proposed method.
  5. [Figure 2] The PCA visualization is qualitative; a quantitative alignment measure, such as mean cosine similarity between aligned Qphi and text embeddings, would strengthen the claim of strong semantic alignment.
  6. [Table 3] Objective metrics are reported without confidence intervals or any measure of variability across the 50 utterances; this is particularly relevant for the small DNSMOS differences and the WER comparison.
  7. [Section 5] The sentence "SemAlignVC achieves the highest naturalness, only comparable to HierSpeech++" is vague; please clarify whether the difference is statistically meaningful.
  8. [Abstract] The word "significantly" in "significantly reduces timbre leakage" is not backed by a statistical significance test; consider rephrasing to avoid implying formal significance.
  9. [Section 3.2] The text says "ensuring that no timbre-related cues remain," which is too strong given that the evidence is a single linear probe and the model still uses a reference mel spectrogram as a timbre condition.

Circularity Check

2 steps flagged · score 3.0 of 10

Internal speaker-removal diagnostics are consequences of SemAlign's training objective, but the external benchmark comparisons are not circular.

  1. self definitional [Section 3.2 (SemAlign training objective) and Section 5 (interpretation of Table 1)]
    "We introduce SemAlign, which uses the text τ of the utterance Xas,at and pass it through a pretrained text-only semantic encoder (BERT in our case) to obtain τs = BERT(τ ) and align this with the output Qϕ using Monotonic Alignment Search (MAS) [34–36]. ... we minimize the mean square error ( Lsem) between the upsampled τs and ˆas. This process forces the semantic encoder to learn only the semantic component while effectively removing any speaker information. ..."

    The 2.84% speaker-classification accuracy of Qϕ is presented as evidence that SemAlign removes timbre, but Qϕ is explicitly trained to approximate BERT text embeddings by minimizing Lsem, and BERT is a text-only encoder with no speaker identity. Low linearly decodable speaker information is therefore a direct consequence of the training target rather than an emergent or independently confirmed property. The probe verifies that the model followed its objective, but it does not provide independent evidence for the disentanglement claim.

  2. self definitional [Section 5, Figure 2 (PCA of Qϕ and τs)]
    "To further analyze the information captured by the semantic encoder Qϕ and compare it with the text-derived embedding τs, we perform a Principal Component Analysis (PCA) on both representations. The results, visualized in Fig. 2, reveal a strong alignment between the principal components of the audio embeddings and their textual counterparts. This alignment further reinforces the effectiveness of our approach in ensuring that the extracted features primarily encode semantic content while minimizing speaker-dependent characteristics."

    The PCA compares Qϕ's output with τs, which is exactly the quantity that Lsem minimizes against during training. Since the method is defined by aligning the audio representation to the text embedding, a high PCA overlap between them is expected by construction. The figure is a sanity check on the training objective, not an independent confirmation that the representation is speaker-independent.

full rationale

The central voice-conversion evaluation is not circular: SemAlignVC is compared with official checkpoints of KNNVC, HierSpeech++, and UniAudio on SMOS, DNSMOS, WER, FPC, and three speaker-similarity embeddings, and those external benchmarks are independent of the method's fitted parameters. The two internal diagnostics that the paper highlights as evidence of timbre removal, namely the near-chance speaker classification of Qϕ and the PCA overlap between Qϕ and τs, reduce to the training objective itself: BERT text embeddings are speaker-independent by construction, and Lsem explicitly pushes Qϕ toward those embeddings. These are self-definitional validations rather than emergent discoveries, which raises the circularity score moderately. There is no load-bearing self-citation chain or imported uniqueness theorem; the MAS and flow-matching machinery are standard external techniques. One additional non-circular concern is that the abstract claims superior intelligibility while Table 3 reports WER 12.31% for SemAlignVC versus 8.24% for HierSpeech++; that is an internal consistency or correctness issue, not a circularity, so it does not affect the circularity score beyond the discussion.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim depends on the assumption that text embeddings are a speaker-free, content-complete target for speech and that monotonic alignment correctly maps them to audio. The paper introduces no new physical entities; all components are pretrained or trained models. The main unstated knobs are the SemAlign loss weight, the reference segment length, and the semantic encoder capacity.

free parameters (3)
  • SemAlign MSE loss weight = not reported
    Strength of text-alignment supervision relative to the LLM cross-entropy loss is not stated; it controls how aggressively timbre is removed versus content preserved.
  • Reference segment fraction = 0.25
    The 25% excerpt of the utterance is used as the timbre reference; it determines how much target speaker information reaches the LLM and acoustic model.
  • Semantic encoder capacity = 4 conformer layers, hidden dim 2048
    Capacity of Qphi determines whether it can retain phonetic detail after being forced to match BERT embeddings; no ablation is reported.
assumptions (6)
  • domain assumption BERT text embeddings are speaker-independent and contain the same semantic content as the spoken utterance.
    Section 3.2: tau_s = BERT(tau) is used as the target for Qphi via MAS and MSE; if BERT encodes style or lacks phonetic detail, timbre removal or content loss follows.
  • domain assumption Text-to-speech alignment is monotonic and MAS with a beta-binomial prior finds the correct alignment between BERT text frames and audio frames.
    Section 3.2: MAS is applied to align tau_s to Qphi; misalignment would inject errors into the semantic representation.
  • domain assumption Utterance-level mean normalization of pitch and energy removes speaker-related cues while preserving paralinguistic information.
    Section 3.2: normalized f0 and energy are fed to the LLM; if normalization removes speaker cues it may also remove prosody, affecting FPC.
  • ad hoc to paper CTC blank tokens retain unwanted speaker characteristics, motivating SemAlign.
    Section 3.2: 'We suspect this is due to the nature of the CTC training inherently allowing blank tokens can retain unwanted speaker characteristics.' This is an untested hypothesis used to justify the new method.
  • domain assumption Speaker classification accuracy on frozen representations is a valid probe of the amount of speaker information.
    Section 4.1: linear classification heads are trained on each representation; the probe assumes a linear head can detect whatever speaker information is linearly accessible.
  • domain assumption Pretrained frozen components, including the audio tokenizer, BERT, and BigVGAN, do not reintroduce timbre leakage or degradations.
    Section 3.2: these are frozen; the paper does not analyze their contribution to timbre or intelligibility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment." pith.science (2026). https://pith.science/paper/QUP4IQW4

@misc{pith2026250709070,
  author       = {Pith},
  title        = {Pith review of: SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUP4IQW4}},
  note         = {Machine review of arXiv:2507.09070}
}
read the original abstract

Zero-shot voice conversion (VC) synthesizes speech in a target speaker's voice while preserving linguistic and paralinguistic content. However, timbre leakage-where source speaker traits persist-remains a challenge, especially in neural codec and LLM-based VC, where quantized representations entangle speaker identity with content. We introduce SemAlignVC, an architecture designed to prevent timbre leakage using SemAlign, a novel method that aligns text and audio representations to ensure speaker-independent semantic encoding. This disentangled representation conditions an autoregressive transformer for high-fidelity conversion without explicit speaker embeddings. Experiments show SemAlignVC significantly reduces timbre leakage, outperforming baselines in speaker timbre similarity, intelligibility, and naturalness, making it a robust, privacy-preserving, and generalizable VC solution. Audio samples can be accessed at https://shivammehta25.github.io/SemAlignVC/

Figures

Figures reproduced from arXiv: 2507.09070 by the authors.

Figure 1
Figure 1. Schematic overview of training semantic LLM (left) and using the entire pipeline for voice conversion (right) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Principal Component Analysis (PCA) comparison of Qϕ (audio) depicted in blue and τs (text) embeddings depicted in red for two utterances. ment techniques, speaker identity can persist in representations intended to be speaker-independent. Second, our model, trained with SemAlign, demonstrates superior timbre removal capa￾bilities, as evidenced by the low classification accuracy. This result indicates that the learne… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 35 canonical work pages

  1. [1]

    This identity includes accent [2], style, emotion [3], and timbre

    Introduction V oice conversion (VC) replicates speaker identity while pre- serving the semantics and paralinguistic features of speech [1]. This identity includes accent [2], style, emotion [3], and timbre. Zero-shot VC, or any-to-any VC, extends this capability to un- seen speakers, posing challenges due to the absence of paired source-target utterances....

    work page

  2. [2]

    SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment

    Background 2.1. Speech Quantization Speech signals are sampled at high rates, generating thousands of data points per second—posing challenges for LLM-based modeling. To manage this complexity, vector quantization (VQ) techniques [25] reduce the data rate by tokenizing au- dio. Common methods include Residual Vector Quantization (RVQ) [4, 5] and K-Nearest...

    work page Pith review arXiv 2025

  3. [3]

    Additionally, we describe two auxiliary objec- tives, the Connectionist Temporal Classification (CTC) loss [31] and the novel SemAlign methodology

    Method In this section, we formalize the voice conversion framework for non-parallel datasets and introduce our approach, which consists of four individually trained components: audio tokenization, a semantic language model (semantic-LLM), an acoustic model, and a vocoder. Additionally, we describe two auxiliary objec- tives, the Connectionist Temporal Cl...

    work page

  4. [4]

    Experiment 4.1. Speaker Classification Using Acoustic Representations To motivate and demonstrate the effectiveness of SemAlign, we experimented using pre-trained acoustic representations, both discrete and continuous and trained a speaker classification head on top. This was done to evaluate the presence of speaker in- formation. We utilized discrete tok...

    work page

  5. [5]

    Results and Discussions From Table 1, we derive two key observations. First, commonly used audio codecs and semantic encoders inherently retain a significant amount of speaker information, which can inadver- tently lead to timbre leakage, thereby degrading the quality of voice conversion. This suggests that without proper disentangle- 3https://github.com/...

    work page

  6. [6]

    Conclusion In this work, we introduced SemAlignVC, an architecture for zero-shot voice conversion that effectively disentangles semantic content from speaker timbre using SemAlign, a novel approach to filter timbre information from input audio and minimize tim- bre leakage for voice conversion. Our evaluation demonstrated that SemAlignVC outperforms basel...

    work page

  7. [7]

    An overview of voice conversion and its challenges: From statistical modeling to deep learning,

    B. Sisman, J. Yamagishi, S. King, and H. Li, “An overview of voice conversion and its challenges: From statistical modeling to deep learning,” IEEE/ACM TASLP, vol. 29, pp. 132–157, 2021

    work page 2021

  8. [8]

    L2-arctic: A non- native english speech corpus,

    G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev- Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non- native english speech corpus,” inInterspeech 2018, 2018, pp. 2783– 2787

    work page 2018

Show all 49 references
  1. [9]

    Emotional voice conver- sion: Theory, databases and esd,

    K. Zhou, B. Sisman, R. Liu, and H. Li, “Emotional voice conver- sion: Theory, databases and esd,”Speech Communication, vol. 137, pp. 1–18, 2022

    2022

  2. [10]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasac- chi, “Soundstream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021

    2021

  3. [11]

    High fidelity neural audio compression,

    A. D´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023

    2023

  4. [12]

    Speechtok- enizer: Unified speech tokenizer for speech large language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language models,” arXiv:2308.16692, 2023

    2023 arXiv

  5. [13]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin et al., “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” arXiv:2403.03100, 2024

    2024 arXiv

  6. [14]

    Uniaudio: An audio founda- tion model toward universal audio generation,

    D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu et al. , “Uniaudio: An audio founda- tion model toward universal audio generation,”arXiv:2310.00704, 2023

    2023 arXiv

  7. [15]

    Lm-vc: Zero- shot voice conversion via speech generation based on language models,

    Z. Wang, Y . Chen, L. Xie, Q. Tian, and Y . Wang, “Lm-vc: Zero- shot voice conversion via speech generation based on language models,” IEEE Signal Processing Letters, 2023

    2023

  8. [16]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv:2412.10117, 2024

    2024 arXiv

  9. [17]

    Autovc: Zero-shot voice style transfer with only autoencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in Proc. of ICML. PMLR, 2019, pp. 5210–5219

    2019

  10. [18]

    Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance,

    Y . Chen, M. Tu, T. Li, X. Li, Q. Kong et al., “Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  11. [19]

    Freevc: Towards high-quality text-free one-shot voice conversion,

    J. Li, W. Tu, and L. Xiao, “Freevc: Towards high-quality text-free one-shot voice conversion,” inICASSP. IEEE, 2023, pp. 1–5

    2023

  12. [20]

    Unsupervised speech decomposition via triple information bottle- neck,

    K. Qian, Y . Zhang, S. Chang, M. Hasegawa-Johnson, and D. Cox, “Unsupervised speech decomposition via triple information bottle- neck,” in Proc. of ICML. PMLR, 2020, pp. 7836–7846

    2020

  13. [21]

    V oice conversion with just nearest neighbors,

    M. Baas, B. van Niekerk, and H. Kamper, “V oice conversion with just nearest neighbors,” in Interspeech, 2023

    2023

  14. [22]

    Any-to-any voice conversion with f0 and timbre disentanglement and novel timbre conditioning,

    S. Kovela, R. Valle, A. Dantrey, and B. Catanzaro, “Any-to-any voice conversion with f0 and timbre disentanglement and novel timbre conditioning,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  15. [23]

    Phonetic posteriorgrams for many- to-one voice conversion without parallel data training,

    L. Sun, K. Li, H. Wang et al., “Phonetic posteriorgrams for many- to-one voice conversion without parallel data training,” inICME. IEEE, 2016, pp. 1–6

    2016

  16. [24]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM trans- actions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  17. [25]

    Streamvc: Real- time low-latency voice conversion,

    Y . Yang, Y . Kartynnik, Y . Li, J. Tanget al., “Streamvc: Real- time low-latency voice conversion,” inICASSP. IEEE, 2024, pp. 11 016–11 020

    2024

  18. [26]

    Make- a-voice: Unified voice synthesis with discrete representation,

    R. Huang, C. Zhang, Y . Wang, D. Yang, L. Liu et al., “Make- a-voice: Unified voice synthesis with discrete representation,” arXiv:2305.19269, 2023

    2023 arXiv

  19. [27]

    Quickvc: Any-to-many voice conversion using inverse short-time fourier transform for faster conversion,

    H. Guo, C. Liu, C. T. Ishi et al., “Quickvc: Any-to-many voice conversion using inverse short-time fourier transform for faster conversion,”arXiv:2302.08296, 2023

    2023 arXiv

  20. [28]

    A comparison of discrete and soft speech units for improved voice conversion,

    B. Van Niekerk, M.-A. Carbonneau, J. Za¨ıdi et al., “A comparison of discrete and soft speech units for improved voice conversion,” in ICASSP. IEEE, 2022, pp. 6562–6566

    2022

  21. [29]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505– 1518, 2022

    2022

  22. [30]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” arXiv:2005.07143, 2020

    2005 arXiv

  23. [31]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyalset al., “Neural discrete representation learning,” Proc. of NeurIPS, vol. 30, 2017

    2017

  24. [32]

    Self-supervised learning with random-projection quantizer for speech recognition,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in Proc. of ICML. PMLR, 2022, pp. 3915–3924

    2022

  25. [33]

    Stargan-vc2: Rethinking conditional methods for stargan-based voice conver- sion,

    T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “Stargan-vc2: Rethinking conditional methods for stargan-based voice conver- sion,” arXiv:1907.12279, 2019

    1907 arXiv

  26. [34]

    Cyclegan- vc2: Improved cyclegan-based non-parallel voice conversion,

    T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “Cyclegan- vc2: Improved cyclegan-based non-parallel voice conversion,” in ICASSP. IEEE, 2019, pp. 6820–6824

    2019

  27. [35]

    Vqmivc: Vector quantization and mutual information-based unsupervised speech representation disentanglement for one-shot voice conver- sion,

    D. Wang, L. Deng, Y . T. Yeung, X. Chen, X. Liuet al., “Vqmivc: Vector quantization and mutual information-based unsupervised speech representation disentanglement for one-shot voice conver- sion,” arXiv:2106.10132, 2021

    2021 arXiv

  28. [36]

    Vec-tok speech: Speech vectorization and tokenization for neural speech generation,

    X. Zhu, Y . Lv, Y . Lei, T. Li, W. He, H. Zhou, H. Lu, and L. Xie, “Vec-tok speech: Speech vectorization and tokenization for neural speech generation,” arXiv:2310.07246, 2023

    2023 arXiv

  29. [37]

    Con- nectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. of ICML, 2006, pp. 369–376

    2006

  30. [38]

    Prosody-controllable spontaneous tts with neural hmms,

    H. Lameris, S. Mehta, G. E. Henter, J. Gustafson, and ´E. Sz´ekely, “Prosody-controllable spontaneous tts with neural hmms,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  31. [39]

    V oiceshop: A unified speech-to-speech framework for identity-preserving zero- shot voice editing,

    P. Anastassiou, Z. Tang, K. Peng, D. Jia, J. Liet al., “V oiceshop: A unified speech-to-speech framework for identity-preserving zero- shot voice editing,” arXiv:2404.06674, 2024

    2024 arXiv

  32. [40]

    One tts alignment to rule them all,

    R. Badlani, A. Ła´ncucki, K. J. Shih, R. Valle, W. Ping, and B. Catanzaro, “One tts alignment to rule them all,” in ICASSP

  33. [41]

    Dnsmos p.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “Dnsmos p.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP, 2022. 6

    2022

  34. [42]

    Neural hmms are all you need (for high-quality attention-free tts),

    S. Mehta, ´E. Sz´ekely, J. Beskow, and G. E. Henter, “Neural hmms are all you need (for high-quality attention-free tts),” in ICASSP

  35. [43]

    7457–7461

    IEEE, 2022, pp. 7457–7461

    2022

  36. [44]

    OverFlow: Putting flows on top of neural transducers for better TTS,

    S. Mehta, A. Kirkland, H. Lameris, J. Beskow,´E. Sz´ekely, and G. E. Henter, “OverFlow: Putting flows on top of neural transducers for better TTS,” in Interspeech, 2023, pp. 4279–4283

    2023

  37. [45]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in The Eleventh International Conference on Learning Representations, 2023

    2023

  38. [46]

    Matcha- tts: A fast tts architecture with conditional flow matching,

    S. Mehta, R. Tu, J. Beskow, ´E. Sz´ekely, and G. E. Henter, “Matcha- tts: A fast tts architecture with conditional flow matching,” in ICASSP. IEEE, 2024, pp. 11 341–11 345

    2024

  39. [47]

    V oice- box: Text-guided multilingual universal speech generation at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokaret al., “V oice- box: Text-guided multilingual universal speech generation at scale,” Proc. of NeurIPS, vol. 36, 2024.5

    2024

  40. [48]

    Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis,

    S.-H. Lee, H.-Y . Choi, S.-B. Kim, and S.-W. Lee, “Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis,” arXiv:2311.12454, 2023

    2023 arXiv

  41. [2022]

    6092–6096

    IEEE, 2022, pp. 6092–6096

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.