REVIEW 30 cited by
Zero-shot Voice Conversion with Diffusion Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Zero-shot voice conversion aims to transform a source speech utterance to match the timbre of a reference speech from an unseen speaker. Traditional approaches struggle with timbre leakage, insufficient timbre representation, and mismatches between training and inference tasks. We propose Seed-VC, a novel framework that addresses these issues by introducing an external timbre shifter during training to perturb the source speech timbre, mitigating leakage and aligning training with inference. Additionally, we employ a diffusion transformer that leverages the entire reference speech context, capturing fine-grained timbre features through in-context learning. Experiments demonstrate that Seed-VC outperforms strong baselines like OpenVoice and CosyVoice, achieving higher speaker similarity and lower word error rates in zero-shot voice conversion tasks. We further extend our approach to zero-shot singing voice conversion by incorporating fundamental frequency (F0) conditioning, resulting in comparative performance to current state-of-the-art methods. Our findings highlight the effectiveness of Seed-VC in overcoming core challenges, paving the way for more accurate and versatile voice conversion systems.
Forward citations
Cited by 30 Pith papers
-
From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data
KNN retrieval on WavLM features constructs synthetic-to-real pairs for training a zero-shot voice conversion model that generalizes across languages when trained exclusively on English.
-
SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing
SpeechEditBench provides seven atomic editing tasks, compositional multi-operation instructions, and an anchor-based protocol yielding target success, preservation success, and joint success metrics; evaluations show ...
-
Poly-SVC: Polyphony-Aware Singing Voice Conversion with Harmonic Modeling
Poly-SVC converts singing voices from polyphonic recordings while keeping melody, lyrics, and harmonies by combining CQT-based pitch extraction with a conditional flow matching diffusion decoder.
-
X-VC: Zero-shot Streaming Voice Conversion in Codec Space
X-VC achieves zero-shot streaming voice conversion via one-step codec-space conversion with dual-conditioning acoustic converter and role-assignment training on generated paired data.
-
From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction
Voice conversion in interactive studies boosts user trust in SpeechLLM responses while automated metrics detect accent-by-gender disparities in alignment and verbosity.
-
A Geometry-Limited Identification Floor and Its Consequences for Voice-Clone Attribution in Professional Voice Actors
Fixed-threshold voice-clone attribution is unreliable on professional voice actors: a re-ranking-resistant misidentification floor persists, and generic encoders falsely implicate enrolled actors for around half of no...
-
A Geometry-Limited Identification Floor and Its Consequences for Voice-Clone Attribution in Professional Voice Actors
On 1,168 professional voice actors, a misidentification floor in speaker embeddings survives calibration, normalization, and discriminative re-ranking, and the same floor makes fixed-threshold voice-clone attribution ...
-
VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion
Pretrained spoofing detectors reach at best 28.98% EER on a new English-Spanish benchmark of 10 LLM-era TTS/VC systems under 10 post-processing conditions, with most near chance.
-
GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech
GRAFT splices a short spoken word sample into a neural codec TTS prompt and uses voice-conversion training so the model copies that pronunciation into any target voice, cutting target-word phoneme error 22-39%.
-
TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech
Introduces DyadEE dataset and TRACE window-level framework using sequences of acoustic embeddings for emotional entrainment detection, reporting 97.01% accuracy when context and relationship information are included.
-
TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech
TRACE detects synthetic emotional entrainment disruption in dyadic speech at up to 93.47% accuracy when conditioned on relationship, using windowed emotion-Whisper sequences on the new DyadEE dataset.
-
ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
ProsoCodec models prosody as a conditional residual in a speech codec via text and speaker prefix conditioning, yielding improved prosody preservation and less timbre leakage in voice conversion experiments.
-
Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
A MoE-enhanced model with conditional distillation reduces speech-NVV EER from 38.93% to 22.66% and speech EER from 13.17% to 9.24% across 10 NVV types.
-
RTCFake: Speech Deepfake Detection in Real-Time Communication
RTCFake is the first large-scale dataset of real-time communication speech deepfakes paired with offline versions, paired with a phoneme-guided consistency learning method that improves cross-platform and noise-robust...
-
How Far Are Video Models from True Multimodal Reasoning?
Current video models succeed on basic understanding but achieve under 25% success on logically grounded generation and near 0% on interactive generation, exposing gaps in multimodal reasoning.
-
Entropy-based Coarse and Compressed Semantic Speech Representation Learning
Predictive entropy from a token-level speech language model finds merge boundaries, producing compressed semantic tokens that keep ASR and translation accuracy at 15 Hz while lowering latency.
-
AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
AugCodec disentangles speech into semantic, speaker, and prosody tokens via tailored data augmentations, achieving 12.5 Hz operation with three streams and outperforming prior codecs on LibriSpeech reconstruction and ...
-
Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Zero-VC applies speaker anonymization as a perturbation to achieve strictly causal zero-lookahead streaming voice conversion by balancing timbre leakage against prosodic utility.
-
Vibrato Expression Control for Singing Voice Conversion with Improving Independent Control
VibE-SVC2 extends prior singing voice conversion work with new modules for independent pitch-style and timbre-style control, claiming better performance and finer controllability than existing methods.
-
From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data
KNN retrieval over WavLM representations creates synthetic source-target pairs from non-parallel data for supervised voice conversion training with a speaker loss, achieving strong results on multilingual test sets de...
-
Universal Speech Content Factorization
A universal least-squares speech-to-content map plus few-second speaker transforms yields open-set, low-rank, timbre-suppressed features competitive for zero-shot VC and TTS.
-
QASA: Quality-Aware Semantic Augmentation for Robust Multimodal Sentiment Analysis
Diffusion-generated video/audio samples weighted by a learned quality scorer are claimed to improve multimodal sentiment analysis on CH-SIMS, CMU-MOSI, and MUStARD.
-
REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers
REF-VC is a zero-shot voice conversion system that random-erases redundant parts of speech-embedding features to stay robust to noise, and uses shortcut-distilled flow matching to convert speech in only four steps.
-
Kimi-Audio Technical Report
Kimi-Audio is an open-source audio foundation model that achieves state-of-the-art results on speech recognition, audio understanding, question answering, and conversation after pre-training on more than 13 million ho...
-
Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR
Three data-centric strategies are studied to improve rare non-verbal vocalization recognition in ASR while preserving lexical accuracy.
-
Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis
Unified guidance framework for Flow Matching speech synthesis achieves nearly 3x faster inference and improved speaker similarity by combining heterogeneous data augmentation with intrinsic model guidance to eliminate...
-
MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
MeanVC 2 introduces future-receptive chunking and a universal timbre token encoder to achieve lower-latency and more robust streaming zero-shot voice conversion than the original MeanVC.
-
Semantic-Aware Ship Detection with Vision-Language Integration
Abstract claims a VLM-plus-adaptive-window framework and a new semantic ship dataset, but the manuscript body is a different voice-timbre paper.
-
AT-ADD: All-Type Audio Deepfake Detection Challenge Evaluation Plan
AT-ADD introduces standardized tracks and datasets for evaluating audio deepfake detectors on speech under real-world conditions and on diverse unknown audio types to promote generalization beyond speech-centric methods.
-
Intelligent Agents with Emotional Intelligence: Current Trends, Challenges, and Future Prospects
A holistic survey of affective computing for intelligent agents covering emotion understanding via multimodal data, affective cognition, emotional expression synthesis, key challenges, and future directions emphasizin...
Discussion (0). Sign in to comment.