Pith. sign in

REVIEW 4 major objections 5 minor 42 references

StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read StarVC is a unified autoregressive framework that generates text tokens before acoustic tokens, and it reports the lowest word and character error rates among the compared voice-conversion systems while matching the best speaker similarity.

desk verdict A clean transfer of the Moshi-style text-then-speech recipe to voice conversion, with credible gains in intelligibility, but the key ablation is confounded because removing text tokens also removes ASR pretraining and the joint ASR objective. read the letter →

arxiv 2506.02414 v1 pith:2MVKW4CH submitted 2025-06-03 cs.MM cs.CLcs.SDeess.AS

Fengjin Li , Jie Wang , Yadong Niu , Yongqing Wang , Meng Meng , Jian Luan , Zhiyong Wu This is my paper
classification cs.MMcs.CLcs.SDeess.AS
keywords voiceconversionjointtextandspeechgenerationautoregressivelanguagemodeltext-firstconditioningspeakerdisentanglementcontentpreservationaudiocodectokensmulti-stagetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that voice conversion improves when the model is forced to predict a written transcription before it generates the converted audio. The proposed system, StarVC, uses a language model that generates tokens one at a time: it emits text tokens first and then eight layers of audio codec tokens, grounding the speech synthesis in an explicit lexical plan rather than in soft semantic vectors. On a held-out set of 400 clean utterances, this design reports the lowest word error rate (6.27%) and character error rate (4.09%) among the systems compared, while its speaker similarity and naturalness scores are at least tied with the strongest alternatives. The same pipeline also outputs its own transcript of the converted speech, with 4.95% word error and 1.51% character error. If the claim holds, explicit text prediction is a structural choice that helps disentangle who is speaking from what is said.

What carries the argument

The load-bearing mechanism is a delayed text-first autoregressive sequence: the model writes one text token, then the first acoustic token, then the next text token, then the next acoustic tokens, using the MusicGen-style delay pattern so every acoustic decision is conditioned on an explicit lexical prefix. Around this sit three frozen components—a semantic encoder taken from a pretrained speech-recognition system, a speaker encoder, and an eight-layer open audio codec—together with trainable adapters and a 24-layer transformer language model. A three-stage training schedule (ASR pretraining, voice-conversion training, then joint ASR-VC training) is what lets the same weights serve both transcription and conversion.

What would settle it

A concrete check would be to compare per-utterance accuracy of the model's generated text with accuracy of its converted audio: if the central claim is right, utterances where the predicted transcript is wrong should also show degraded acoustic WER, and corrupting the text prediction should propagate into the converted speech; if the two are decoupled, the text step is not the load-bearing mechanism.

Watch

Extended reading notes

Core claim

StarVC's central claim is that inserting an explicit text-generation step into an autoregressive voice-conversion pipeline improves both content preservation and speaker transfer. The model takes semantic features from a frozen speech-recognition encoder and a single speaker embedding from a frozen speaker encoder, then autoregressively produces a sequence of text tokens followed by acoustic tokens from an eight-layer neural audio codec. Because the text tokens are scheduled one step ahead of the first acoustic token, the audio stream is conditioned on an already-decided transcript rather than on latent semantic vectors. In the reported experiments this mechanism delivers the best intelligibility among the compared systems, with 6.27% word error rate and 4.09% character error rate on converted speech, while speaker similarity is effectively on par with the strongest baseline (SECS 0.835 and 0.472 versus 0.839 and 0.478) and subjective similarity and naturalness are the highest (SMOS 3.98, NMOS 4.17). The same pipeline emits its own transcript at 4.95% word error and 1.51% character error, which the paper reads as evidence that the text tokens themselves are accurate.

Load-bearing premise

The system depends on its frozen pretrained components being the right carriers: the speech-recognition encoder must preserve exact words in its semantic features, the speaker encoder must capture the target voice from one reference clip, and the eight-layer audio codec must retain enough detail for natural reconstruction; if any of these representations is mismatched with the conversion objective, the text-conditioning benefit shrinks.

Editorial extensions

If this is right

  • Voice conversion systems built this way can output a readable transcript as a natural byproduct, removing the need for a separate post-hoc ASR pass for captioning or dialogue logging.
  • Because acoustic tokens are decoded from an explicit word sequence, content preservation is less dependent on source prosody and timbre, which may reduce timbre leakage in cross-speaker transfers.
  • The ablations imply that multi-stage training and a sufficiently large language model are needed for the benefit; smaller or single-stage versions lose much of the intelligibility gain.
  • The text-token stream gives the model a direct handle for correction: a listener can repair a mis-transcribed word and have the audio resynthesized with the corrected content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same text-first design could transfer to speech-to-speech translation or zero-shot text-to-speech, where lexical fidelity matters as much as timbre transfer; the architecture does not depend on the voice-conversion setup specifically.
  • Because the generated-text error (4.95% WER) is lower than the converted-audio error (6.27% WER), a portion of the remaining intelligibility loss is probably introduced after text prediction, inside the acoustic codec or decoder; studying which acoustic tokens deviate would locate the bottleneck.
  • The comparison is against baselines that do not generate text; a fair boundary test would be to compare StarVC with a text-aware diffusion or semantic-token voice conversion system to see whether the advantage comes from text prediction itself or from the specific codec and training data.
  • The MOS test uses 20 listeners and 20 source-target pairs; a larger listening panel would be needed to confirm that the small speaker-similarity gap (3.98 versus 3.97) is perceptible rather than noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes StarVC, an autoregressive voice-conversion framework that generates text tokens before acoustic tokens, using a Whisper-small encoder for semantic features, an ERes2Net-large encoder for target-speaker features, a Qwen2.5-style language model, and Mimi codec layers for reconstruction. Training is split into three stages: ASR pretraining, VC training, and joint ASR-VC training, with additional data augmentation from OpenVoice V2. On 400 LibriTTS test-clean utterances, StarVC reports the best WER (6.27%), CER (4.09%), and MOS scores among the compared methods, while also producing text tokens with WER-Text 4.95% and CER-Text 1.51%. The central design claim is that explicitly predicting text before acoustic tokens improves linguistic consistency and speaker preservation.

Significance. The idea of inserting an explicit text-token prediction step into an autoregressive voice-conversion pipeline is timely and potentially useful, especially as speech-text foundation models become more common. If the mechanism were cleanly isolated, the paper would offer a practical recipe for unifying ASR and VC in one model. Strengths of the reported study include a reasonably detailed architecture description, a multi-metric evaluation protocol, an audio demo, and the reporting of both objective and subjective results. Weaker points are the confounded ablation behind the central claim, an abstract-level claim about speaker similarity that Table 1 does not support, and the absence of controlled comparisons that could separate the text-conditioning mechanism from multi-task ASR training. The paper is empirical rather than theoretical; none of the reported losses are circular with the evaluation metrics, which are computed with external tools (Whisper-large, Resemblyzer, WavLM).

major comments (4)
  1. [Abstract and Table 1] The abstract and Section 3.3.1 claim that StarVC outperforms conventional VC methods in preserving speaker characteristics, but Table 1 shows CosyVoice achieving higher SECS-Res (0.839 vs. 0.835) and SECS-Wavlm (0.478 vs. 0.472). The claim is true only for the subjective SMOS in Table 2. Please restrict the speaker-characteristic claim to the metrics that actually support it, or provide confidence intervals or significance tests showing that the small SECS differences are not meaningful.
  2. [Section 3.3.3 and Section 2.2] The 'w/o text token' ablation is confounded. In the described training recipe, removing the text-token output also removes Stage 1 ASR pretraining (L_ASR = CE(y_t, ŷ_t)) and the ASR component of Stage 3 joint training. Therefore the observed WER increase from 6.27% to 7.30% and the SECS-Res drop from 0.835 to 0.771 cannot be attributed specifically to text-conditioned acoustic generation; they may equally reflect the loss of ASR pretraining or multi-task regularization. Since the central claim is that explicit text prediction drives the gains, please add an ablation that retains the ASR objectives (or at least the ASR pretraining stage) while not conditioning acoustic tokens on predicted text, or clearly report a 'w/o text token but with ASR pretraining' condition.
  3. [Section 3.2 and Table 1] The comparison to CosyVoice, OpenVoice V2, and TriAAN-VC is not a controlled test of the text-token mechanism, because these systems differ in architecture, codec, training data, and objective functions. The paper cites LM-VC and DualVC3 in the introduction but does not include them in the evaluation. To support the claim that explicit text prediction is what improves content preservation, the authors should either include LM-based VC baselines that are closer in architecture or rely on within-model ablations that isolate the text-conditioning component.
  4. [Section 2.2 and Table 1] The loss weights w and w' and the schedule of the multi-stage training are never specified, and the 'w/o multi-stage' ablation is not defined precisely. Without knowing what remains in that ablation (only VC training? only joint training? which loss weights?) and what the exact coefficients are, the claimed contribution of multi-stage training cannot be assessed. Please report the numeric values of w and w', the definition of the 'w/o multi-stage' condition, and the training configuration of the 'smaller model' row.
minor comments (5)
  1. [Section 2.1.2] The delay method from MusicGen is mentioned but not precisely specified: it would help to state the exact delay pattern used between text tokens and the eight acoustic codebook layers, and to clarify how subword text tokenization interacts with frame-level acoustic tokens.
  2. [Section 2.3] The ratio between real and OpenVoice V2-synthesized parallel data is reported as 50%/50% in VC training and 80%/20% in joint training, but it is not stated whether the same target speaker can appear with both real and synthesized references or how the model handles speaker-identity conflicts; a sentence clarifying the data construction would be useful.
  3. [Section 3.2] The evaluation set is limited to 400 LibriTTS test-clean utterances with 200 source and 200 target utterances; this is acceptable for a conference paper, but the authors should note that the test set is a single-domain clean English condition and that the generalization to noisy or multilingual speech is untested.
  4. [Table 1] The table format for WER and CER columns (e.g., '8.24% / 4.27%') is ambiguous: it appears to show two metrics in one column. Separating WER and CER into two clear columns would improve readability.
  5. [Section 3.3.2] The MOS confidence intervals are reported, but no significance tests are provided for the objective metrics. Given that some SECS differences are small (e.g., 0.835 vs. 0.839), a paired test or bootstrap confidence intervals would strengthen the objective claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: StarVC is an empirical system comparison with external metrics; the text-token ablation is confounded but not circular.

full rationale

This is an empirical systems paper, not a derivation. The central design — predicting text tokens before acoustic tokens — is implemented and tested, and all reported target metrics are measured with external tools (Whisper-large, Resemblyzer, fine-tuned WavLM, and human listeners) that were not used to set training constants. No metric is fitted by construction: WER-Text and CER-Text directly report the accuracy of the model's generated text tokens, which are optimized by a cross-entropy loss but not by minimizing the evaluation metrics themselves. The paper contains no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via the authors' own prior work. The one substantive concern is the 'w/o text token' ablation in Section 3.3.3, which removes not only text-token conditioning but also the ASR pretraining stage and the joint ASR loss; the observed WER/SECS changes therefore cannot be uniquely attributed to text conditioning. This is an experimental-control limitation, not circularity, because the ablation's outcome is not equivalent to its input by definition, and the central comparison against external baselines remains independently meaningful.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a set of frozen pretrained models and several hand-chosen hyperparameters. No new physical or theoretical entities are introduced. The pretrained components carry most of the semantic, speaker, and acoustic prior knowledge; if any of them underperforms in a target domain, the reported gains may not generalize. The hyperparameters (loss weights, data ratios, task mix) are set without sensitivity analysis, so their optimality is unknown.

free parameters (4)
  • loss balance weight w = not specified in paper
    Weight between text loss and acoustic loss in the VC training loss (Section 2.2.2). Its value is chosen by hand and affects the trade-off between text accuracy and audio quality.
  • acoustic layer loss weights lambda_i = [1.0, 1.0, 0.9, 0.9, 0.8, 0.8, 0.7, 0.7]
    Hand-set weights across the eight Mimi codebook layers (Section 3.1). These balance reconstruction of semantic versus fine acoustic details, influencing speaker similarity and naturalness.
  • real vs synthesized target data ratio = 50/50 in VC training, 80/20 in joint training
    Ratio of real LibriTTS/GigaSpeech targets to OpenVoice V2-synthesized targets (Section 2.3, 3.1). This is a data-augmentation knob adjusted by hand.
  • ASR vs VC task allocation in joint training = 20% ASR, 80% VC
    The proportion of training batches assigned to ASR versus VC objectives (Section 2.2.3). Chosen without sensitivity analysis.
assumptions (5)
  • domain assumption Whisper-small's frozen encoder provides semantic features that retain exact linguistic content for downstream text prediction.
    Invoked in Section 2.1.1; if the encoder mis-recognizes or loses words, the text anchor and subsequent acoustic generation would inherit those errors.
  • domain assumption ERes2Net-large's frozen speaker embedding, given one reference clip, captures transferable speaker identity for any target speaker.
    Invoked in Section 2.1.1; the quality of speaker transfer depends entirely on this embedding's discriminativeness.
  • domain assumption Mimi codec with eight layers can reconstruct speech naturalness and speaker timbre with acceptable fidelity for conversion.
    Invoked in Section 2.1.3; if codec artifacts are present, both MOS and SECS scores would be inflated or deflated relative to true audio quality.
  • domain assumption The MusicGen delay pattern, teacher forcing, and RoPE allow the language model to condition acoustic tokens on prior text tokens without a train-inference mismatch.
    Used in Section 2.1.2; exposure bias or positional mismatch would degrade the claimed text-acoustic grounding.
  • domain assumption Evaluation metrics (Whisper-large WER, Resemblyzer/WavLM SECS, listener MOS) are reliable proxies for intelligibility and speaker similarity.
    Invoked in Section 3.2; these metrics are external and standard, but no calibration or human-adjudication of WER errors is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion." pith.science (2026). https://pith.science/paper/2MVKW4CH

@misc{pith2026250602414,
  author       = {Pith},
  title        = {Pith review of: StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MVKW4CH}},
  note         = {Machine review of arXiv:2506.02414}
}
read the original abstract

Voice Conversion (VC) modifies speech to match a target speaker while preserving linguistic content. Traditional methods usually extract speaker information directly from speech while neglecting the explicit utilization of linguistic content. Since VC fundamentally involves disentangling speaker identity from linguistic content, leveraging structured semantic features could enhance conversion performance. However, previous attempts to incorporate semantic features into VC have shown limited effectiveness, motivating the integration of explicit text modeling. We propose StarVC, a unified autoregressive VC framework that first predicts text tokens before synthesizing acoustic features. The experiments demonstrate that StarVC outperforms conventional VC methods in preserving both linguistic content (i.e., WER and CER) and speaker characteristics (i.e., SECS and MOS). Audio demo can be found at: https://thuhcsi.github.io/StarVC/.

Figures

Figures reproduced from arXiv: 2506.02414 by the authors.

Figure 1
Figure 1. Architecture of StarVC 2. Proposed Approach 2.1. System Architecture Our proposed StarVC framework is designed to jointly model speech conversion and text generation in an auto-regressive manner. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of Feature Extractors are concatenated and input into the LM (Language Model) to generate the output sequence: Y = LM(S, S′ ) = {yˆt, yˆa1 , . . . , yˆan }, yˆt ∈ R Vt , yˆai ∈ R Va where n is the number of the codebook layers, Vt and Va rep￾resent the vocabulary sizes for text and acoustic tokens respec￾tively. 2.1.3. Acoustic Decoder For waveform reconstruction, we utilize Mimi3 , an open-source audio… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages

  1. [1]

    V oice Conversion (VC) is a technique that modifies the speaker identity of an utterance while preserving its linguistic content

    Introduction Speaker identity is one of the fundamental characteristics of hu- man speech, influencing communication, personalization and speech perception. V oice Conversion (VC) is a technique that modifies the speaker identity of an utterance while preserving its linguistic content. Traditional VC methods [1, 2] often at- tempt to extract and manipulat...

    arXiv 2025

  2. [2]

    System Architecture Our proposed StarVC framework is designed to jointly model speech conversion and text generation in an auto-regressive manner

    Proposed Approach 2.1. System Architecture Our proposed StarVC framework is designed to jointly model speech conversion and text generation in an auto-regressive manner. As shown in Figure 1, the system consists of four main components: Semantic Feature Extractor, Speaker Feature Ex- tractor, Language Model and Acoustic Decoder. 2.1.1. Feature Extractors ...

    work page

  3. [3]

    Experiments 3.1. Training We train our model on 8 NVIDIA H100 80G GPUs using three sequential phases: ASR Pretraining (30 hours), VC Training 4https://github.com/myshell-ai/OpenVoice (50 hours) and Joint ASR-VC Training (100 hours). Our LM is a 24-layer transformer [27] (intermediate size=4,864, embed- ding dim=896 and 14 heads), following the same archit...

    work page

  4. [4]

    Conclusion In this work, we introduced StarVC, a multi-stage voice con- version framework that integrates speech generation with text prediction, ensuring stronger linguistic consistency. By lever- aging large-scale labeled data (GigaSpeech, LibriTTS) and an autoregressive modeling approach, StarVC enhances the dis- entanglement of speaker identity and li...

    work page

  5. [5]

    Acknowledgements This work is supported by National Natural Science Founda- tion of China (62076144) and Shenzhen Science and Technol- ogy Program (JCYJ20220818101014030)

  6. [6]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,”arXiv preprint arXiv:1904.05862, 2019

    arXiv 1904

  7. [7]

    Continuous probabilis- tic transform for voice conversion,

    Y . Stylianou, O. Capp´e, and E. Moulines, “Continuous probabilis- tic transform for voice conversion,”IEEE Transactions on speech and audio processing, vol. 6, no. 2, pp. 131–142, 1998

    work page 1998

  8. [8]

    Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion

    A. R. Bargum, S. Serafin, and C. Erkut, “Reimagining speech: A scoping review of deep learning-powered voice conversion,” arXiv preprint arXiv:2311.08104, 2023

    work page Pith review arXiv 2023

Show all 42 references
  1. [9]

    An overview of voice conversion and its challenges: From statistical modeling to deep learning,

    B. Sisman, J. Yamagishi, S. King, and H. Li, “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 132–157, 2020

    2020

  2. [10]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  3. [11]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  4. [12]

    Openvoice: Versatile instant voice cloning,

    Z. Qin, W. Zhao, X. Yu, and X. Sun, “Openvoice: Versatile instant voice cloning,”arXiv preprint arXiv:2312.01479, 2023

    2023 arXiv

  5. [13]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  6. [14]

    Neural codec language mod- els for disentangled and textless voice conversion,

    A. Baade, P. Peng, and D. Harwath, “Neural codec language mod- els for disentangled and textless voice conversion,” inProc. Inter- speech 2024, 2024, pp. 182–186

    2024

  7. [15]

    Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,

    J. Li, Y . Guo, X. Chen, and K. Yu, “Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 296– 12 300

    2024

  8. [16]

    Diffusion-based voice conversion with fast maximum likelihood sampling scheme,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, M. Kudinov, and J. Wei, “Diffusion-based voice conversion with fast maximum likelihood sampling scheme,”arXiv preprint arXiv:2109.13821, 2021

    2021 arXiv

  9. [17]

    Sta- blevc: Style controllable zero-shot voice conversion with condi- tional flow matching,

    J. Yao, Y . Yan, Y . Pan, Z. Ning, J. Ye, H. Zhou, and L. Xie, “Sta- blevc: Style controllable zero-shot voice conversion with condi- tional flow matching,”arXiv preprint arXiv:2412.04724, 2024

    2024 arXiv

  10. [18]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  11. [19]

    Lm-vc: Zero- shot voice conversion via speech generation based on language models,

    Z. Wang, Y . Chen, L. Xie, Q. Tian, and Y . Wang, “Lm-vc: Zero- shot voice conversion via speech generation based on language models,”IEEE Signal Processing Letters, 2023

    2023

  12. [20]

    Dualvc 3: Leveraging language model generated pseudo con- text for end-to-end low latency streaming voice conversion,

    Z. Ning, S. Wang, P. Zhu, Z. Wang, J. Yao, L. Xie, and M. Bi, “Dualvc 3: Leveraging language model generated pseudo con- text for end-to-end low latency streaming voice conversion,”arXiv preprint arXiv:2406.07846, 2024

    2024 arXiv

  13. [21]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  14. [22]

    Mini-omni: Language models can hear, talk while thinking in streaming,

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,”arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  15. [23]

    Llama- omni: Seamless speech interaction with large language models,

    Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “Llama- omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  16. [24]

    Snac: Multi- scale neural audio codec,

    H. Siuzdak, F. Gr ¨otschla, and L. A. Lanzend ¨orfer, “Snac: Multi- scale neural audio codec,”arXiv preprint arXiv:2410.14411, 2024

    2024 arXiv

  17. [25]

    An enhanced res2net with local and global feature fusion for speaker verification,

    Y . Chen, S. Zheng, H. Wang, L. Cheng, Q. Chen, and J. Qi, “An enhanced res2net with local and global feature fusion for speaker verification,”arXiv preprint arXiv:2305.12838, 2023

    2023 arXiv

  18. [26]

    Simple and controllable music gen- eration,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music gen- eration,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  19. [27]

    A learning algorithm for contin- ually running fully recurrent neural networks,

    R. J. Williams and D. Zipser, “A learning algorithm for contin- ually running fully recurrent neural networks,”Neural computa- tion, vol. 1, no. 2, pp. 270–280, 1989

    1989

  20. [28]

    Roformer: Enhanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: Enhanced transformer with rotary position embedding,”Neuro- computing, vol. 568, p. 127063, 2024

    2024

  21. [29]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  22. [30]

    GigaSpeech covers diverse domains and acoustic conditions, while LibriTTS is cleaner and more consistent

    (10,000 hours) and LibriTTS [31, 32] (960 hours) . GigaSpeech covers diverse domains and acoustic conditions, while LibriTTS is cleaner and more consistent. By combining them, we enable StarVC to handle both real-world variability and well-segmented, high-quality speech. 3.2. ...

  23. [31]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    2024

  24. [32]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu, X. Xu, Y . Yuan, M. Wu, W. Wang, and M. D. Plumb- ley, “Semanticodec: An ultra low bitrate semantic audio codec for general sound,”arXiv preprint arXiv:2405.00233, 2024

    2024 arXiv

  25. [33]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,”Advances in Neural In- formation Processing Systems, 2017

    2017

  26. [34]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  27. [35]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  28. [36]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,

    G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhanget al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021

    2021 arXiv

  29. [37]

    Libritts: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,”arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  30. [38]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  31. [39]

    Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Maet al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  32. [40]

    Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wanget al., “Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,”arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  33. [41]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” inInternational Con- ference on Machine Learning. PMLR, 2022, pp. 2709–2720

    2022

  34. [42]

    Triaan- vc: Triple adaptive attention normalization for any-to-any voice conversion,

    H. J. Park, S. W. Yang, J. S. Kim, W. Shin, and S. W. Han, “Triaan- vc: Triple adaptive attention normalization for any-to-any voice conversion,” inICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.