Pith. sign in

REVIEW 3 major objections 5 minor 19 cited by

Zero-shot Voice Conversion with Diffusion Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that zero-shot voice conversion improves when training perturbs the source timbre and inference conditions on the full reference utterance, beating OpenVoice and CosyVoice on both speaker similarity and intelligibility.

desk verdict Useful VC result with a real methodological gap: the paper never says whether inference uses the timbre shifter, and the headline ablation of that shifter is deferred to future work. read the letter →

arxiv 2411.09943 v1 pith:2Q6L6V7L submitted 2024-11-15 cs.SD cs.LGeess.AS

Songting Liu This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords zero-shotvoiceconversiondiffusiontransformertimbreleakageexternalshifterin-contextlearningflowmatchingsingingspeakersimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Seed-VC, a zero-shot voice conversion system that transforms a source utterance to sound like an unseen target speaker from a short reference clip. Its central claim is that two design choices resolve the usual trade-off between timbre similarity and intelligibility: perturbing the source timbre with an external timbre shifter during training, and conditioning a diffusion transformer on the entire reference utterance rather than on a single timbre vector. Seed-VC is reported to outperform existing baselines on both speaker similarity and word error rate in zero-shot voice conversion, and to extend to singing voice conversion with pitch conditioning. If correct, the framework offers a way to reduce timbre leakage without the information loss caused by bottleneck or discretization methods.

What carries the argument

The mechanism is the external timbre shifter in the training loop, realized here by an imperfect voice-conversion model, combined with a flow-matching diffusion transformer. The shifter maps source audio $X_{\text{src}}$ to $X_{\text{shifted}}=T(X_{\text{src}}, e_r)$, and semantic features are extracted from both utterances; during training, the target portion of the noisy acoustic sequence is conditioned on shifted semantic features while the prompt portion uses original semantic features and original acoustics. The model is trained by minimizing a flow-matching $L^1$ loss on the predicted velocity field, with context $c=[e_{\text{timbre}}, S]$ formed from a speaker-verification timbre vector and the semantic sequence. At inference, the prompt is the full reference utterance, letting the transformer do in-context timbre learning. U-Net-style skip connections, time-as-token embeddings, and rotary positional embeddings are the supporting architectural choices.

What would settle it

Take the trained Seed-VC and re-run training with the timbre shifter disabled, so semantic features always come from the original source; if speaker similarity and word error rate stay roughly unchanged, the shifter is not doing the claimed leakage removal. A more direct test is to feed source utterances and their shifted versions into the semantic encoder and classify the resulting features with a speaker-verification model: if the shifted features still identify the original speaker at high accuracy, the premise that shifting removes timbre fails.

Watch

Extended reading notes

Core claim

The discovery Seed-VC argues for is that the training objective for zero-shot voice conversion should be a reconstruction from deliberately mismatched inputs: the model must denoise the original acoustics using content features extracted from a timbre-shifted version of the same utterance, plus a timbre vector and prompt acoustics from the original reference. Because the content features come from speech that no longer carries the original timbre, the model cannot lean on source identity and is forced to treat the semantic stream as content-only; because the reference is supplied as a full utterance rather than a single vector, the transformer can attend to fine-grained timbre cues. The paper reports that this combination beats the baselines on speaker similarity while also lowering word error rate, and that ablating the full-reference enrollment degrades similarity.

Load-bearing premise

The load-bearing premise is that running the source utterance through an imperfect timbre-shifting model removes enough of the source speaker's timbre from the extracted semantic features that the model learns content that is genuinely speaker-neutral, while the timbre shifter's own errors do not corrupt the linguistic content.

Editorial extensions

If this is right

  • Zero-shot voice conversion can be trained to reconstruct original acoustics from shifted content plus reference timbre, so the model never sees matched content and timbre pairs and the training-inference gap shrinks.
  • Semantic encoders that retain residual speaker information can be used without discretization or bottlenecking, preserving linguistic detail and lowering word error rate.
  • Full-reference enrollment transfers the in-context learning benefits already seen in zero-shot text-to-speech to voice conversion, improving speaker similarity.
  • The same training recipe, with F0 conditioning and gender-based pitch shifts, applies to singing voice conversion with competitive pitch accuracy and better lyric preservation than a strong baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shifter's only job is to strip enough source timbre, any voice-conversion or semantic-to-acoustic model that reliably changes speaker identity could substitute for the one used here, so the method should be robust to the choice of shifter.
  • A testable extension would be to measure residual speaker identity directly in the shifted semantic features and use that measurement to choose or train the shifter, rather than taking the shifter's output as sufficient.
  • The same shifted-content training signal could be applied to text-to-speech or speech editing pipelines, where matched content and timbre pairs are the norm and timbre leakage is less often addressed.
  • The gender-based pitch shift used for singing conversion is a rule of thumb; future work could replace it with a learned or reference-derived F0 mapping for cross-gender singing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Seed-VC, a zero-shot voice conversion system combining a flow-matching diffusion transformer with a training-time external timbre shifter (OpenVoice V2). During training, a random segment of the source utterance is used as an acoustic and semantic prompt, while the model denoises the remaining segment conditioned on semantic features extracted from a timbre-shifted version of the source, together with a speaker embedding taken from the source. At inference, the full reference utterance is intended to serve as the timbre enrollment. The paper reports experiments on LibriTTS/Emilia-101k showing higher speaker similarity and lower WER than OpenVoice and CosyVoice, and a singing-voice extension with F0 conditioning that compares favorably with RVCv2 on M4Singer.

Significance. If the reported gains hold, Seed-VC would be a useful contribution to zero-shot voice conversion: it combines full-reference in-context timbre conditioning with a training-time leakage-reduction strategy, and the code and pretrained models are released. The paper's strengths include direct comparisons against two strong open-source baselines, an ablation row isolating full-reference enrollment (Table 1), and a singing-voice extension. However, the central mechanism, the timbre shifter, is not ablated, and the inference-time use of the shifter is left unspecified, so the significance of the headline result is conditional on additional evidence.

major comments (3)
  1. [Section 3.3.2, Eqs. (6)-(10), Figures 2-3] The training procedure feeds the target/noise portion of the acoustic sequence with semantic features S_shifted = f_semantic(T(X_src, e_r)) (Eq. 8), while the prompt portion uses S_src and original acoustic features A. At inference, Figure 3 is only described as corresponding to the training pipeline, and the text never states whether the source utterance is passed through the timbre shifter before semantic extraction. If inference uses unshifted source semantics, the model receives content features in the target role that were never seen during training, so the claimed alignment of training and inference is not realized and the Table 1 gains cannot be attributed to the timbre-shifter mechanism. If inference does apply T, this step must be stated and evaluated, because it makes OpenVoice a component of the proposed system rather than only a baseline.
  2. [Section 5 and Introduction contribution bullet; Table 1] The introduction claims the paper provides 'insights through ablation studies on the impact of the timbre shifter, the use of full reference speech context, and different timbre shifter methods,' but the only ablation reported is 'Seed-VC (without full reference enrollment)' in Table 1. Section 5 explicitly defers ablation studies for the timbre shifter to future work. Because the timbre shifter is the core innovation behind the leakage-mitigation claim, omitting its ablation leaves the headline comparison uninterpretable: the gains in Table 1 could come from full-reference in-context conditioning, from the Emilia-101k training scale, or from using OpenVoice as the shifter rather than from leakage reduction. A minimal fix is to ablate the shifter (e.g., train with S_src instead of S_shifted in the target role) and to report WER/SECS of the shifted utterances to validate the premise that T preserves content while removing source timbre.
  3. [Section 4.3.1, Table 1] The paper reports mean SECS, WER, CER, and DNSMOS values over 100 source utterances and 8 target utterances, but no error bars, confidence intervals, or significance tests are provided. Statements such as 'Seed-VC significantly outperforms' (Abstract and Section 4.3.1) are therefore not supported by the reported evidence; for example, the SECS gap between Seed-VC and CosyVoice (0.8676 vs. 0.8440) is small, and the WER gap could be sensitive to evaluation variance. The authors should report per-item standard errors or bootstrap confidence intervals, or at minimum state the number of trials per condition and the variance across source/target pairs.
minor comments (5)
  1. [Title page] The affiliation line contains 'Coumputing', and the text contains many broken tokens such as 'OpenV oice', 'V oice', and 'Y et' that appear to be LaTeX ligature/space artifacts; these should be cleaned in revision.
  2. [Section 3.1, Eq. (1)] The text introduces a source flow f_s and a target flow f_t, but only f_s appears in the loss; the role of the target flow should be clarified or the notation corrected.
  3. [Section 3.2 and Section 4.1.1] There are several typos: 'perpended' should be 'prepended', 'embeeding' should be 'embedding', and 'ablility' should be 'ability'.
  4. [Table 2] The table header 'FOCORR↑ FORMSE↓' should be 'F0CORR' and 'F0RMSE' to match the text, and the text should consistently use the same metric abbreviations.
  5. [Section 4.3.2] The text refers to 'Section 3.4' for the gender-based pitch shift adjustment, but this material appears in Section 3.5.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical model comparisons with independent external metrics; no fitted metric is renamed as a prediction and no load-bearing self-citation chain exists.

full rationale

The derivation chain is not circular. Seed-VC's training objective L_FM (Eq. 1) is a standard flow-matching reconstruction of acoustic features conditioned on c = [etimbre, S] (Eq. 4). The timbre shifter (Eq. 6) is an input augmentation that produces X_shifted, and S_shifted = f_semantic(X_shifted) (Eq. 8) is used only as a training-time content condition for the target/noise segment; at no point is the evaluation SECS/WER/CER used as a training loss or fitted parameter. The comparison with OpenVoice/CosyVoice is an external benchmark, and SECS uses Resemblyzer, WER/CER uses HuBERT-ASR, and DNSMOS uses Microsoft's predictor—all independent of the model's internal Whisper/CAM++ features. There is also no self-citation load-bearing: the paper invokes no uniqueness theorem from the authors' prior work; the only cited prior models (AutoVC, FreeVC, YourTTS/OpenVoice, CosyVoice, Whisper, CAM++, BigVGAN, RMVPE) are external tools. One genuine weakness is that Section 3.3.2 trains the target component with S_shifted while Figure 3/3.4 do not explicitly state that inference passes the source through T before semantic extraction; if inference uses S_src, the claim that the timbre shifter 'aligns training with inference' is not demonstrated. This is a train/inference consistency limitation (and the paper defers the ablation in Section 5), but it is not a circular reduction of a predicted quantity to an input. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The core speech conversion claim rests on standard speech components (Whisper, CAM++, BigVGAN, OpenVoice, flow matching) and a few domain assumptions rather than on fitted free parameters. The only hand-set scalar in the paper is the pitch shift magnitude used in the singing extension.

free parameters (1)
  • Gender pitch shift magnitude = +12 / -12 semitones
    Hand-set heuristic in Section 3.5 for male-to-female and female-to-male singing conversion; not fitted to data but used in the singing extension.
assumptions (5)
  • domain assumption Whisper semantic features are speaker-invariant enough to serve as content-only conditioning.
    Invoked in Section 3.3.2 equations (7) and (8), where Ssrc and Sshifted are used as linguistic content without discretization or speaker normalization.
  • domain assumption OpenVoice timbre shifter alters timbre while preserving linguistic content.
    Assumed in Section 3.3.1, where equation (6) defines Xshifted and the shifted audio is treated as a valid source of content features.
  • domain assumption The full reference audio at inference provides transferable timbre through in-context conditioning.
    Used in Section 3.4, where the entire reference audio is included as model input without an explicit mechanism to separate its content from its timbre.
  • domain assumption Objective metrics (SECS, WER, DNSMOS) are valid proxies for conversion quality.
    Assumed throughout Section 4.1.3; no human listening test is reported.
  • domain assumption Training and evaluation speakers do not overlap, despite both using public web-derived data.
    Assumed in Section 4.1.2 when claiming zero-shot conditions; Emilia-101k may contain LibriTTS or seed-tts-eval speakers, and no overlap check is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-shot Voice Conversion with Diffusion Transformers." pith.science (2026). https://pith.science/paper/2Q6L6V7L

@misc{pith2026241109943,
  author       = {Pith},
  title        = {Pith review of: Zero-shot Voice Conversion with Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Q6L6V7L}},
  note         = {Machine review of arXiv:2411.09943}
}
read the original abstract

Zero-shot voice conversion aims to transform a source speech utterance to match the timbre of a reference speech from an unseen speaker. Traditional approaches struggle with timbre leakage, insufficient timbre representation, and mismatches between training and inference tasks. We propose Seed-VC, a novel framework that addresses these issues by introducing an external timbre shifter during training to perturb the source speech timbre, mitigating leakage and aligning training with inference. Additionally, we employ a diffusion transformer that leverages the entire reference speech context, capturing fine-grained timbre features through in-context learning. Experiments demonstrate that Seed-VC outperforms strong baselines like OpenVoice and CosyVoice, achieving higher speaker similarity and lower word error rates in zero-shot voice conversion tasks. We further extend our approach to zero-shot singing voice conversion by incorporating fundamental frequency (F0) conditioning, resulting in comparative performance to current state-of-the-art methods. Our findings highlight the effectiveness of Seed-VC in overcoming core challenges, paving the way for more accurate and versatile voice conversion systems.

Figures

Figures reproduced from arXiv: 2411.09943 by the authors.

Figure 1
Figure 1. Architectural Detail about U-Net style skip connections and timestamp condition. 3.1 Flow Matching We adopt a flow matching diffusion scheme for training and inference. The flow matching algorithm aligns distributions by minimizing the discrepancy between a source flow fs and a target flow ft. Given a source distribution ps(x) and a target distribution pt(x), the algorithm constructs a continuous path by learning a … view at source ↗
Figure 2
Figure 2. Training Pipeline. A random segment is set as timbre prompt. Prompt component contains semantic feature and acoustic feature from original audio, while target component contains semantic feature from timbre-shifted audio. Loss is only calculated on target component. 3.3 Addressing Timbre Leakage and Training-Inference Gap To mitigate timbre leakage and align the training process with the inference scenario, we intro… view at source ↗
Figure 3
Figure 3. Inference Pipeline. Corresponds to training pipeline, reference audio plays the role as timbre enrollment. 3.4 Enhanced Timbre Representation As in Section 3.3.2, a random audio segment is used as context and excluded from loss calculation. This allows the model to capture more fine-grained timbre information besides timbre vector. During inference process shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

    cs.CV 2026-08 conditional novelty 6.0 of 10

    UniSwap is a streaming audio-video diffusion model that transfers a reference face and voice onto a source talking video while keeping motion, scene, and speech content.

  2. Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

    cs.SD 2026-08 conditional novelty 6.0 of 10

    A jointly trained speech tokenizer and flow-matching decoder achieve strong zero-shot TTS and voice conversion on English and Mandarin, with WER below ground truth in several tests.

  3. A Geometry-Limited Identification Floor and Its Consequences for Voice-Clone Attribution in Professional Voice Actors

    eess.AS 2026-07 conditional novelty 6.0 of 10

    On 1,168 professional voice actors, a misidentification floor in speaker embeddings survives calibration, normalization, and discriminative re-ranking, and the same floor makes fixed-threshold voice-clone attribution ...

  4. VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Pretrained spoofing detectors reach at best 28.98% EER on a new English-Spanish benchmark of 10 LLM-era TTS/VC systems under 10 post-processing conditions, with most near chance.

  5. GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech

    cs.LG 2026-07 accept novelty 6.0 of 10

    GRAFT splices a short spoken word sample into a neural codec TTS prompt and uses voice-conversion training so the model copies that pronunciation into any target voice, cutting target-word phoneme error 22-39%.

  6. TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    TRACE detects synthetic emotional entrainment disruption in dyadic speech at up to 93.47% accuracy when conditioned on relationship, using windowed emotion-Whisper sequences on the new DyadEE dataset.

  7. Speaker Anonymisation for Speech-based Suicide Risk Detection

    eess.AS 2025-09 conditional novelty 6.0 of 10

    Combining voice-conversion and text-to-speech anonymisation preserves suicide-risk detection accuracy near the original baseline (0.692 vs 0.702) while raising speaker-verification EER toward chance.

  8. An Extensive Analysis of the Singing Voice Conversion Challenge 2025 Evaluation Results

    cs.SD 2025-09 conditional novelty 6.0 of 10

    SVCC 2025 shows top systems can match ground truth singer identity but cannot yet match naturalness or singing style, with breathy, glissando, and vibrato as the hardest styles.

  9. Entropy-based Coarse and Compressed Semantic Speech Representation Learning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Predictive entropy from a token-level speech language model finds merge boundaries, producing compressed semantic tokens that keep ASR and translation accuracy at 15 Hz while lowering latency.

  10. SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods

    cs.SD 2025-07 conditional novelty 6.0 of 10

    SpeechFake is a large-scale multilingual deepfake speech dataset with baseline experiments showing improved generalization to unseen generation methods.

  11. The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents

    cs.CR 2025-07 unverdicted novelty 6.0 of 10

    A multi-agent audio-language model framework can automatically profile private attributes, such as age, health, and income, directly from general audio recordings.

  12. Universal Speech Content Factorization

    eess.AS 2026-03 conditional novelty 5.0 of 10

    A universal least-squares speech-to-content map plus few-second speaker transforms yields open-set, low-rank, timbre-suppressed features competitive for zero-shot VC and TTS.

  13. QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysis

    cs.LG 2026-01 reject novelty 5.0 of 10

    Diffusion-generated video/audio samples weighted by a learned quality scorer are claimed to improve multimodal sentiment analysis on CH-SIMS, CMU-MOSI, and MUStARD.

  14. REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

    eess.AS 2025-08 unverdicted novelty 5.0 of 10

    REF-VC is a zero-shot voice conversion system that random-erases redundant parts of speech-embedding features to stay robust to noise, and uses shortcut-distilled flow matching to convert speech in only four steps.

  15. De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks

    cs.SD 2025-07 conditional novelty 5.0 of 10

    Existing voice-protection perturbations succeed only against naive attackers; a phoneme-guided purification-refinement pipeline restores cloneability of protected speech for most VC models.

  16. IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

    cs.CL 2025-06 conditional novelty 5.0 of 10

    IndexTTS2 achieves precise token-count-based duration control and emotion/speaker disentanglement in an autoregressive zero-shot TTS, reporting SOTA WER, speaker similarity, and emotional fidelity.

  17. Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

    cs.SD 2026-08 conditional novelty 4.0 of 10

    Confucius4-TTS performs transcript-free, cross-lingual zero-shot voice cloning in 14 languages with competitive intelligibility and speaker similarity.

  18. Semantic-Aware Ship Detection with Vision-Language Integration

    cs.CV 2025-08 unverdicted novelty 4.0 of 10

    Abstract claims a VLM-plus-adaptive-window framework and a new semantic ship dataset, but the manuscript body is a different voice-timbre paper.

  19. EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion

    cs.SD 2025-05 reject novelty 4.0 of 10

    EZ-VC combines discrete units from a multilingual self-supervised encoder (Xeus) with an F5-TTS flow-matching decoder to achieve zero-shot any-to-any voice conversion, without text labels or multiple disentangling encoders.

Reference graph

Works this paper leans on

42 extracted references · 14 canonical work pages · cited by 19 Pith papers

  1. [1]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  2. [2]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  3. [3]

    Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,

    L. Sun, K. Li, H. Wang, S. Kang, and H. Meng, “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,” in 2016 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2016, pp. 1–6

    work page 2016

  4. [4]

    Autovc: Zero-shot voice style transfer with only autoencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in International Conference on Machine Learning. PMLR, 2019, pp. 5210–5219

    2019

  5. [5]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al., “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017

    2017

  6. [6]

    V oice conversion with just nearest neighbors,

    M. Baas, B. van Niekerk, and H. Kamper, “V oice conversion with just nearest neighbors,” arXiv preprint arXiv:2305.18975, 2023

    arXiv 2023

  7. [7]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020

    2020

  8. [8]

    Diffwave: A versatile diffusion model for audio synthesis,

    Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” arXiv preprint arXiv:2009.09761, 2020

    arXiv 2009

Show all 42 references
  1. [9]

    Wavegrad: Estimating gradients for waveform generation,

    N. Chen, Y . Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, “Wavegrad: Estimating gradients for waveform generation,” arXiv preprint arXiv:2009.00713, 2020

    2009 arXiv

  2. [10]

    Diffusion-based voice conversion with fast maximum likelihood sampling scheme,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, M. Kudinov, and J. Wei, “Diffusion-based voice conversion with fast maximum likelihood sampling scheme,” arXiv preprint arXiv:2109.13821, 2021

    2021 arXiv

  3. [11]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017. 12 A PREPRINT - NOVEMBER 18, 2024

    2017

  4. [12]

    Diffsinger: Singing voice synthesis via shallow diffusion mechanism,

    J. Liu, C. Li, Y . Ren, F. Chen, and Z. Zhao, “Diffsinger: Singing voice synthesis via shallow diffusion mechanism,” in Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 10, 2022, pp. 11 020–11 028

    2022

  5. [13]

    Streamvc: Real-time low-latency voice conversion,

    Y . Yang, Y . Kartynnik, Y . Li, J. Tang, X. Li, G. Sung, and M. Grundmann, “Streamvc: Real-time low-latency voice conversion,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 016–11 020

    2024

  6. [14]

    Diff-hiervc: Diffusion-based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,

    H.-Y . Choi, S.-H. Lee, and S.-W. Lee, “Diff-hiervc: Diffusion-based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,” International Speech Communication Association, pp. 2283–2287, 2023

    2023

  7. [15]

    Mullivc: Multi-lingual voice conversion with cycle consistency,

    J. Huang, C. Zhang, Y . Ren, Z. Jiang, Z. Ye, J. Liu, J. He, X. Yin, and Z. Zhao, “Mullivc: Multi-lingual voice conversion with cycle consistency,”arXiv preprint arXiv:2408.04708, 2024

    2024 arXiv

  8. [16]

    Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion,

    H.-Y . Choi, S.-H. Lee, and S.-W. Lee, “Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17 862–17 870

    2024

  9. [17]

    Ppg-based singing voice conversion with adversarial representation learning,

    Z. Li, B. Tang, X. Yin, Y . Wan, L. Xu, C. Shen, and Z. Ma, “Ppg-based singing voice conversion with adversarial representation learning,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7073–7077

    2021

  10. [18]

    Leveraging content-based features from multiple acoustic models for singing voice conversion,

    X. Zhang, Y . Gu, H. Chen, Z. Fang, L. Zou, L. Xue, and Z. Wu, “Leveraging content-based features from multiple acoustic models for singing voice conversion,”arXiv preprint arXiv:2310.11160, 2023

    2023 arXiv

  11. [19]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv preprint arXiv:2104.00355, 2021

    2021 arXiv

  12. [20]

    Freevc: Towards high-quality text-free one-shot voice conversion,

    J. Li, W. Tu, and L. Xiao, “Freevc: Towards high-quality text-free one-shot voice conversion,” inICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  13. [21]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2023, pp. 4195–4205

    2023

  14. [22]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  15. [23]

    Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer,

    K. Lee, D. W. Kim, J. Kim, and J. Cho, “Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer,” arXiv preprint arXiv:2406.11427, 2024

    2024 arXiv

  16. [24]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,

    S. E. Eskimez, X. Wang, M. Thakker, C. Li, C.-H. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tanet al., “E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,”arXiv preprint arXiv:2406.18009, 2024

    2024 arXiv

  17. [25]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  18. [26]

    Openvoice: Versatile instant voice cloning,

    Z. Qin, W. Zhao, X. Yu, and X. Sun, “Openvoice: Versatile instant voice cloning,”arXiv preprint arXiv:2312.01479, 2023

    2023 arXiv

  19. [27]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Ma et al., “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  20. [28]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,

    K. Shen, Z. Ju, X. Tan, Y . Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” arXiv preprint arXiv:2304.09116, 2023

    2023 arXiv

  21. [29]

    V oicecraft: Zero-shot speech editing and text-to-speech in the wild,

    P. Peng, P.-Y . Huang, S.-W. Li, A. Mohamed, and D. Harwath, “V oicecraft: Zero-shot speech editing and text-to-speech in the wild,” arXiv preprint arXiv:2403.16973, 2024

    2024 arXiv

  22. [30]

    All are worth words: A vit backbone for diffusion models,

    F. Bao, S. Nie, K. Xue, Y . Cao, C. Li, H. Su, and J. Zhu, “All are worth words: A vit backbone for diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 22 669–22 679

    2023

  23. [31]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. S...

    2015

  24. [32]

    Enhanced transformer with rotary position embedding., 2021,

    J. Su, Y . Lu, S. Pan, A. Murtadha, B. Wen, and Y . L. Roformer, “Enhanced transformer with rotary position embedding., 2021,” DOI: https://doi. org/10.1016/j. neucom, 2023

    2021 doi

  25. [33]

    Cam++: A fast and efficient network for speaker verification using context-aware masking,

    H. Wang, S. Zheng, Y . Chen, L. Cheng, and Q. Chen, “Cam++: A fast and efficient network for speaker verification using context-aware masking,” arXiv preprint arXiv:2303.00332, 2023. 13 A PREPRINT - NOVEMBER 18, 2024

    2023 arXiv

  26. [34]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 2709–2720

    2022

  27. [35]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” arXiv preprint arXiv:2407.05361, 2024

    2024 arXiv

  28. [36]

    Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6493–6497

    2021

  29. [37]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  30. [38]

    Bigvgan: A universal neural vocoder with large-scale training,

    S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “Bigvgan: A universal neural vocoder with large-scale training,” arXiv preprint arXiv:2206.04658, 2022

    2022 arXiv

  31. [39]

    M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus,

    L. Zhang, R. Li, S. Wang, L. Deng, J. Liu, Y . Ren, J. He, R. Huang, J. Zhu, X. Chen et al., “M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus,” Advances in Neural Information Processing Systems, vol. 35, pp. 6914–6926, 2022

    2022

  32. [40]

    Rmvpe: A robust model for vocal pitch estimation in polyphonic music,

    H. Wei, X. Cao, T. Dan, and Y . Chen, “Rmvpe: A robust model for vocal pitch estimation in polyphonic music,” arXiv preprint arXiv:2306.15412, 2023

    2023 arXiv

  33. [41]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in International Conference on Machine Learning. PMLR, 2021, pp. 5530–5540

    2021

  34. [42]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms,

    T. SpeechTeam, “Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms,” arXiv preprint arXiv:2407.04051, 2024. 14

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.