Pith. sign in

REVIEW 4 major objections 5 minor 50 references

Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RV-TTS generates speech whose voice is chosen by a face image and whose style is set by descriptive text, producing multiple plausible voices per face and keeping a chosen voice consistent via prompting.

desk verdict Strong face-driven TTS system with a real quality jump, but the audio-only branch has an unstated leakage risk that needs to be resolved before the headline ablation claim holds. read the letter →

arxiv 2505.18972 v1 pith:P643TLW4 submitted 2025-05-25 eess.AS cs.AI

Minsu Kim , Pingchuan Ma , Honglie Chen , Stavros Petridis , Maja Pantic This is my paper
classification eess.AScs.AI
keywords text-to-speechsynthesisface-drivenTTSmulti-modalspeechcodelanguagemodelcontrastivelearningstyletransfervoiceconsistencycontrollable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RV-TTS is a text-to-speech model that takes three inputs: the text to be spoken, a face image that determines the voice, and a natural-language description that controls speech characteristics such as pace, noise level, distance, tone, and recording place. The paper argues that previous face-driven TTS systems sound noisy, fail on artistic portraits, and give inconsistent voices, so it adds three mechanisms: training on clean audio-only speech alongside audio-visual data, random style transfer on face images, and a sampling-then-prompting procedure that makes voice generation both diverse and repeatable. The result, the paper reports, is speech that is substantially more natural and more face-matching than earlier face-driven systems, with 76 percent speaker-identification accuracy on real faces and 73 percent on artistic portraits.

What carries the argument

The generative core is an autoregressive Transformer that predicts 9-level residual vector-quantized (RVQ) audio codes in a delayed pattern; a pre-trained language model (T5) encodes the descriptive text, and a face encoder (ResNet50 initialized with a face-recognition model) supplies a face-driven voice embedding. The voice embedding and text embedding are concatenated along the time axis and passed into cross-attention layers. To align the two modalities, the face encoder and an audio encoder (ECAPA-TDNN) are pre-trained with an InfoNCE contrastive loss so that face-driven and audio-driven embeddings for the same speaker occupy the same latent space; the TTS model is then trained by alternating between the two embedding types depending on whether the batch comes from audio-visual or audio-only data. Style augmentation applies random neural style transfer, grayscaling, and blurring to the input face to close the gap between real photos and artistic portraits. At inference, top-k sampling produces diverse candidate voices, and a generated sample is reused as a prompt to keep the voice consistent across sentences.

What would settle it

Train a variant of RV-TTS in which the audio-driven embedding is always extracted from a separate, non-overlapping reference utterance of the target speaker (never the target speech itself), and compare MOS/FMS against the reported numbers; if scores fall sharply, the original result was at least partly explained by content leakage.

Watch

Extended reading notes

Core claim

The central claim is that a speech-code language model can be conditioned on a shared voice-embedding space built by contrastively aligning a face encoder with an audio encoder, so the model alternates between face-driven and audio-driven embeddings during training. This lets a high-quality audio-only corpus (LibriTTS-R) teach clean, stuttering-free synthesis while the audio-visual corpus (LRS3 and VoxCeleb2) teaches the face-to-voice association. Style augmentation with neural style transfer extends the same association to paintings, sketches, and old photos. Because decoding is sampling-based, the model can imagine several voices for one face; feeding one generated sample back as a prompt then locks the voice for subsequent utterances. The paper reports that this design outperforms prior face-driven TTS systems in human-rated naturalness (MOS 4.14 vs 1.84) and face matching (FMS 3.86 vs 2.31), and that descriptive-text changes move the intended acoustic metrics in the expected direction.

Load-bearing premise

The audio-only training branch conditions the model on an embedding drawn from an utterance of the target speaker, and the paper does not state whether that utterance is the target speech itself or a separate reference, so the reported gains could come partly from content leakage on LibriTTS-R rather than from the face-to-voice mapping.

Editorial extensions

If this is right

  • A user can generate several candidate voices from a single face, pick a preferred one, and then generate any number of sentences with that same voice by prompting with the chosen sample.
  • Descriptive text shifts specific acoustic properties in the expected direction: words like 'slow' lower speaking rate, 'monotone' reduces pitch variance, and 'distant' reduces C50 clarity.
  • Including high-quality audio-only speech corpora during training improves both naturalness and face-voice matching, meaning future face-driven TTS need not be limited to noisy audio-visual recordings.
  • Style augmentation extends the model to paintings, sketches, and old photographs, so historical figures known only through artwork can be given a plausible voice.
  • On human evaluations the model improves MOS by 2.30 and FMS by 1.45 over the strongest prior face-driven baseline, and reaches 76% and 73% speaker-identification accuracy on real faces and artistic portraits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the audio-only branch's voice embedding is extracted from the same utterance that serves as the target speech, the reported gains on LibriTTS-R might come partly from content leakage; a cleaner diagnostic would use a separate reference utterance from the same speaker and re-measure MOS/FMS.
  • The sampling-then-prompting design separates voice exploration from voice fixation, which suggests it could be combined with other conditioning signals, such as an external reference voice or a text description of the desired voice, to fine-tune the final output.
  • The controllability analysis reports aggregate acoustic shifts, so a listener study is needed to confirm that naive users actually perceive differences such as 'moderate distant' versus 'very distant' in the intended way.
  • Because the face and audio encoders are initialized from recognition models trained on large face and speaker datasets, the aligned voice-embedding space may carry their demographic biases; evaluating on diverse faces across ages, ethnicities, and artistic styles would test how broadly the generalization holds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RV-TTS, a multi-modal controllable text-to-speech system that generates speech whose voice characteristics are conditioned on a face image, while speech traits (pace, tone, noise, distance, place) are controlled by a natural-language description. The method combines a speech code language model (following MusicGen/Parler-TTS) with a contrastively pre-trained face/audio encoder pair, style augmentation to handle artistic portraits, and a training scheme that alternates between face-driven and audio-driven voice embeddings to leverage high-quality audio-only data (LibriTTS-R). The authors claim three technical contributions: using audio-only corpora via alternating voice embeddings, stylization augmentation for robustness to non-photorealistic faces, and sampling-based decoding with prompting to obtain diverse yet consistent voices. Experiments report large gains over prior face-driven TTS methods in subjective MOS, face-matching score (FMS), and speaker identification accuracy, alongside a controllability analysis with descriptive text.

Significance. If the reported results hold, the paper makes a substantial contribution to face-driven TTS, addressing realistic limitations of prior work that suffers from noisy audio and poor generalization to artistic portraits. The reported improvements over FaceTTS and FVTTS in MOS (4.14 vs 1.84/1.60) and FMS (3.86 vs 2.31/2.38) are large, and the 76% and 73% speaker identification accuracies on LRS3 and artistic portraits are convincing. The use of human raters for face-voice matching is an independent, credible evaluation. The idea of alternating face- and audio-driven conditioning to exploit clean audio-only data is interesting and potentially useful for the community. However, several methodological details and ablations need to be clarified or strengthened before these claims are fully supported.

major comments (4)
  1. [§2.3, §3.2] The paper does not specify whether the sample utterance ca used for the audio-driven conditioning is the target speech y or a different utterance of the same speaker. In the contrastive pre-training stage a random 3–5s segment is drawn, but no analogous rule is stated for the RV-TTS training. If ca = y, the audio-driven embedding za would contain the exact content and prosody of the target, allowing the autoregressive code language model to copy the target and invalidating the LibriTTS-R quality gain in Table 1 as evidence for face-driven clean-speech synthesis. Additionally, the ECAPA-TDNN encoder is initialized from MFCC-trained weights but its input is modified to accept RVQ codes; the paper states the encoder is frozen except the attached linear layer, yet provides no explanation or validation that a frozen MFCC-trained trunk produces meaningful embeddings from RVQ-code inputs. Please specify whether ca is the target or a separate reference utterance, and provide an experimental validation of the modified audio encoder (e.g., speaker-verification accuracy on VoxCeleb2 test or an ablation comparing ca=y vs ca≠y).
  2. [Table 1, §3.4.1] The ablation for contrastive learning is confounded. The row labeled 'replacing contrastive learning' (✗ ✓ ✗) differs from the full model (✓ ✓ ✓) in two factors: it removes both HQ audio data and contrastive learning. The text claims that 'the use of contrastive learning primarily contributes to preserving face-voice association' based on the FMS and SIM drops, but those drops could be partly due to the removal of HQ audio. The clean comparison isolating contrastive learning would be row 2 (✗ ✓ ✓) versus row 4 (✗ ✓ ✗), since both lack HQ audio. Please report this comparison or add an ablation that holds HQ audio constant.
  3. [§2.4, §3.4] The claimed one-to-many capability—that sampling-based decoding produces multiple plausible voices for a single face—is not quantitatively evaluated anywhere in the paper. The VCS metric measures consistency across multiple generations, but there is no experiment showing that the sampled voices are actually distinct or diverse (e.g., distribution of predicted speaker embeddings, human judgment of distinct speaker identities, or a diversity metric). Since this is a stated contribution (Contribution 4), please add an experiment that validates the diversity of the generated voices from the same face.
  4. [§3.3, Table 1, Table 2] The evaluation protocol is inconsistent across the paper. The metrics section states that MOS and FMS are measured from 20 samples of LRS3, but Table 1 reports '30 samples of both LRS3 and artistic portraits,' and Section 3.4.2 says results on LRS3 and artistic portraits are shown in Table 2 without specifying the composition of the test set. Please clarify exactly which test set each table reports, the number of samples, and why the sample sizes differ between the ablation and the main comparison. This is necessary for the reader to interpret the scores and the claimed improvements correctly.
minor comments (5)
  1. [§2.2] The sentence 'the image is augmented with a 50% chance among one of the following: style transferring, gray-scaling, and blurring, each with a uniform probability' is ambiguous: it could mean each of the three augmentations is applied independently with 50% probability, or that with 50% probability exactly one of the three is chosen uniformly. Please clarify the exact augmentation schedule.
  2. [§3.2] The paper states that the ECAPA-TDNN input is 'modified with RVQ codes instead of MFCCs' but does not specify how discrete RVQ code indices are converted into a continuous representation. State whether code embeddings are concatenated, averaged, or otherwise transformed before feeding into the frozen trunk.
  3. [Table 1] The SIM scores in Table 1 are 0.11, 0.11, 0.11, and 0.09. With the reported small evaluation set (30 samples), these differences appear likely to be within noise. Please report confidence intervals or a significance test for the SIM comparisons, particularly for the claimed degradation when contrastive learning is removed.
  4. [§3.4.2] Parler-TTS is a random-voice synthesis method not conditioned on a face, so the comparison in Table 2 is not apples-to-apples; consider describing it as a reference point or an upper/lower bound rather than a direct competitor, to avoid implying a fair face-driven comparison.
  5. [§3.4.3] The controllability analysis in Table 3 relies on synthetic descriptive labels generated by Data-Speech for LRS3 and VoxCeleb2, but the paper does not state whether the evaluation uses the same generated labels or held-out annotations, nor whether the measured metric values (e.g., SI-SDR, C50) are averaged over multiple random generations. Please clarify the protocol and report variability (e.g., standard deviation) for the feature values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central face-to-voice results rest on external human evaluation, and the only self-citations are off-the-shelf tools.

full rationale

The paper's central claim is that RV-TTS can synthesize speech whose voice matches a face image and whose characteristics follow descriptive text. The derivation chain is: (1) contrastive pretraining aligns face and audio embeddings; (2) RV-TTS is trained with either face- or audio-driven embeddings as conditions; (3) style augmentation bridges portraits; (4) sampling plus prompting gives diverse yet consistent voices. None of these steps reduces to its inputs by construction. The MOS, FMS, and speaker-identification evaluations are external human judgments, not functions of the training labels. The only self-citations are the use of Auto-AVSR [36] for VoxCeleb2 transcriptions and [5] in the introduction; both are off-the-shelf tools with independent published evaluation and do not supply the central claim. The controllability analysis measures output speech with independent estimators (Speaking Rate, SI-SDR, C50, pitch std) rather than reading back the conditioning text. One potential self-definitional leak would occur if the audio-only conditioning utterance ca were the target utterance y itself, since the model could then copy target content from the conditioning embedding; the paper defines ca as a separate 'sample utterance' and does not state that ca = y, so this remains an unvalidated implementation detail rather than an exhibited circular step. No circular step can be quoted from the paper.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the learnability of face-voice associations from paired video, the interchangeability of face and audio embeddings after contrastive learning, and the content-invariance of the audio encoder; none of these is verified outside the paper's own experiments.

assumptions (3)
  • domain assumption Face images carry sufficient information to infer a plausible voice, and this association generalizes to artistic portraits after style augmentation.
    The entire task is defined by this assumption; §1 and §2.2 rely on it.
  • domain assumption A pretrained ECAPA-TDNN audio encoder modified to accept RVQ codes still produces a speaker-discriminative, content-invariant embedding.
    The audio-only training branch (§2.3) depends on this to avoid content leakage, but no validation is provided.
  • domain assumption Contrastive learning aligns face and audio embeddings well enough that the TTS model can treat them as interchangeable conditions.
    The alternation in §2.3 requires this interchangeability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis." pith.science (2026). https://pith.science/paper/P643TLW4

@misc{pith2026250518972,
  author       = {Pith},
  title        = {Pith review of: Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P643TLW4}},
  note         = {Machine review of arXiv:2505.18972}
}
read the original abstract

This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image, and the characteristics of output speech (e.g., pace, noise level, distance, tone, place) can be controllable with natural text description. Specifically, we aim to mitigate the following three challenges in face-driven TTS systems. 1) To overcome the limited audio quality of audio-visual speech corpora, we propose a training method that additionally utilizes high-quality audio-only speech corpora. 2) To generate voices not only from real human faces but also from artistic portraits, we propose augmenting the input face image with stylization. 3) To consider one-to-many possibilities in face-to-voice mapping and ensure consistent voice generation at the same time, we propose to first employ sampling-based decoding and then use prompting with generated speech samples. Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.

Figures

Figures reproduced from arXiv: 2505.18972 by the authors.

Figure 1
Figure 1. Overview of the proposed RV-TTS: The face image controls the voice, the descriptive text controls speech charac￾teristics, and the input text determines the content of speech. high-quality face-driven TTS model; 1) The data often con￾tains noisy audio and includes stuttering, whereas clean TTS data [14, 15] is essential for high-quality TTS training. 2) The data is based on real humans, which means that the model tr… view at source ↗
Figure 2
Figure 2. Illustration of the proposed RV-TTS. (a) The face image is randomly stylized using a pre-trained style transfer model to reduce the gap between real human faces and artistic portraits. (b) The face encoder and audio encoder are pre-trained through contrastive learning to share a common embedding space. (c) During training, the model alternates between face-driven and audio-driven voice embeddings to learn not only t… view at source ↗
Figure 3
Figure 3. Speaker identification test results comparisons. face-driven TTS methods, the proposed RV-TTS outperforms previous methods. In order to better understand how the generated speech match the provided face image, we perform a speaker identi￾fication test, where participants are asked to select one face im￾age out of two that best matches the generated speech. A total of 78 samples were rated by 15 people. Identificatio… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 45 canonical work pages

  1. [1]

    Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis

    Introduction Text-to-Speech Synthesis (TTS) is one of the widely used tech- nologies in diverse real-world applications [1–6]. With recent developments, synthesized audio has become increasingly real- istic and natural, and even the voice of outputs can be cloned with a few seconds of sample voices [7, 8]. In this paper, we explore an interesting extensio...

    work page Pith review arXiv 2025

  2. [2]

    Method Let x be input text which will be read by the TTS model, y be the ground-truth speech, ci be the face image, ca be the sam- ple utterance, and ct be the descriptive text. The objective of our learning problem is to synthesize speech y that reads the in- put text x with the voice characteristics of the conditioned face image ci, while also capturing...

    work page

  3. [3]

    Dataset We employ audio-visual speech dataset, LRS3 and V oxCeleb2, and audio-only high-quality speech dataset, LibriTTS-R, to train RV-TTS

    Experiments 3.1. Dataset We employ audio-visual speech dataset, LRS3 and V oxCeleb2, and audio-only high-quality speech dataset, LibriTTS-R, to train RV-TTS. For evaluation, we employ the test set of LRS3 and artistic portraits, a collection of 20 copyright-free images. LRS3 [13] consists of over 400 hours of video collected from TED and TEDx talks. It co...

    work page

  4. [4]

    By employing style augmentation during training, the model can generate voices from artistic portraits as well as normal hu- man face images

    Conclusion We proposed RV-TTS, a multi-modal controllable TTS model. By employing style augmentation during training, the model can generate voices from artistic portraits as well as normal hu- man face images. By alternating voice embeddings from differ- ent modalities, the model successfully learns to associate faces with voices and generates high-quali...

    work page

  5. [5]

    Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss et al., “Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions,” in ICASSP. IEEE, 2018, pp. 4779–4783

    work page 2018

  6. [6]

    Parallel tacotron: Non- autoregressive and controllable tts,

    I. Elias, H. Zen, J. Shen et al. , “Parallel tacotron: Non- autoregressive and controllable tts,” in ICASSP. IEEE, 2021, pp. 5709–5713

    work page 2021

  7. [7]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan et al., “Fastspeech 2: Fast and high-quality end-to-end text to speech,” in ICLR, 2021

    work page 2021

  8. [8]

    Prompttts: Controllable text-to- speech with text descriptions,

    Z. Guo, Y . Leng, Y . Wu et al., “Prompttts: Controllable text-to- speech with text descriptions,” inICASSP. IEEE, 2023, pp. 1–5

    work page 2023

Show all 50 references
  1. [9]

    Textless unit-to-unit train- ing for many-to-many multilingual speech-to-speech translation,

    M. Kim, J. Choi, D. Kim et al. , “Textless unit-to-unit train- ing for many-to-many multilingual speech-to-speech translation,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024

    2024

  2. [10]

    Naturalspeech: End-to-end text- to-speech synthesis with human-level quality,

    X. Tan, J. Chen, H. Liu et al., “Naturalspeech: End-to-end text- to-speech synthesis with human-level quality,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  3. [11]

    Adaspeech 4: Adaptive text to speech in zero-shot scenarios,

    Y . Wu, X. Tan, B. Liet al., “Adaspeech 4: Adaptive text to speech in zero-shot scenarios,” in Interspeech, 2022

    2022

  4. [12]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu et al., “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  5. [13]

    Face2speech: Towards multi- speaker text-to-speech synthesis using an embedding vector pre- dicted from a face image

    S. Goto, K. Onishi, Y . Saito et al., “Face2speech: Towards multi- speaker text-to-speech synthesis using an embedding vector pre- dicted from a face image.” in INTERSPEECH, 2020, pp. 1321– 1325

    2020

  6. [14]

    Synthe-sees: Face based text-to-speech for virtual speaker,

    J. H. Park, J.-G. Maeng, T. Bak et al., “Synthe-sees: Face based text-to-speech for virtual speaker,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2024, pp. 10 321–10 325

    2024

  7. [15]

    Mm-tts: Multi-modal prompt based style transfer for expressive text-to-speech synthesis,

    W. Guan, Y . Li, T. Li et al., “Mm-tts: Multi-modal prompt based style transfer for expressive text-to-speech synthesis,” inProceed- ings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, 2024, pp. 18 117–18 125

    2024

  8. [16]

    Lip reading sentences in the wild,

    J. S. Chung, A. Senior, O. Vinyals et al., “Lip reading sentences in the wild,” in CVPR, 2017, pp. 3444–3450

    2017

  9. [17]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  10. [18]

    Libritts-r: A restored multi- speaker text-to-speech corpus,

    Y . Koizumi, H. Zen, S. Karita et al., “Libritts-r: A restored multi- speaker text-to-speech corpus,” in Interspeech, 2023

    2023

  11. [19]

    Mls: A large-scale multilin- gual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram et al., “Mls: A large-scale multilin- gual dataset for speech research,” in Interspeech, 2020

    2020

  12. [20]

    Imaginary voice: Face- styled diffusion model for text-to-speech,

    J. Lee, J. S. Chung, and S.-W. Chung, “Imaginary voice: Face- styled diffusion model for text-to-speech,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  13. [21]

    Fvtts: Face based voice synthesis for text-to-speech,

    M. Lee, E. Park, and S. Hong, “Fvtts: Face based voice synthesis for text-to-speech,” in Interspeech, 2024, pp. 4953–4957

    2024

  14. [22]

    Image style transfer using convolutional neural networks,

    L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in CVPR, 2016, pp. 2414– 2423

    2016

  15. [23]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  16. [24]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs et al., “High-fidelity audio compression with improved rvqgan,” Advances in Neural Infor- mation Processing Systems, vol. 36, 2024

    2024

  17. [25]

    Natural language guidance of high- fidelity text-to-speech with synthetic annotations,

    D. Lyth and S. King, “Natural language guidance of high- fidelity text-to-speech with synthetic annotations,” arXiv preprint arXiv:2402.01912, 2024

    2024 arXiv

  18. [26]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat et al., “Simple and controllable music generation,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  19. [27]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar et al., “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  20. [28]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren et al., “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778

    2016

  21. [29]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” in Interspeech, 2020

    2020

  22. [30]

    Domain enhanced arbi- trary image style transfer via contrastive learning,

    Y . Zhang, F. Tang, W. Dong et al. , “Domain enhanced arbi- trary image style transfer via contrastive learning,” in ACM SIG- GRAPH, 2022, pp. 1–8

    2022

  23. [31]

    Libritts: A corpus derived from librispeech for text-to-speech,

    H. Zen, V . Dang, R. Clark et al., “Libritts: A corpus derived from librispeech for text-to-speech,” in Interspeech, 2019

    2019

  24. [32]

    Miipher: A robust speech restoration model integrating self-supervised speech and text rep- resentations,

    Y . Koizumi, H. Zen, S. Karita et al., “Miipher: A robust speech restoration model integrating self-supervised speech and text rep- resentations,” in WASPAA. IEEE, 2023

    2023

  25. [33]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  26. [34]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi et al., “A simple framework for contrastive learning of visual representations,” inInternational conference on machine learning. PMLR, 2020, pp. 1597–1607

    2020

  27. [35]

    Language models are few- shot learners,

    T. Brown, B. Mann, N. Ryder et al., “Language models are few- shot learners,” Advances in neural information processing sys- tems, vol. 33, pp. 1877–1901, 2020

    1901

  28. [36]

    Data-speech,

    Y . Lacombe, V . Srivastav, and S. Gandhi, “Data-speech,” https: //github.com/ylacombe/dataspeech, 2024

    2024

  29. [37]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” in Interspeech, 2018

    2018

  30. [38]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia et al. , “Learning audio-visual speech representation by masked multimodal cluster prediction,” in ICLR, 2022

    2022

  31. [39]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu et al., “Robust speech recognition via large-scale weak supervision,” in ICML. PMLR, 2023, pp. 28 492–28 518

    2023

  32. [40]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez et al. , “Auto-avsr: Audio-visual speech recognition with automatic labels,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  33. [41]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue et al., “Arcface: Additive angular margin loss for deep face recognition,” in CVPR, 2019, pp. 4690–4699

    2019

  34. [42]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” in ICLR, 2015

    2015

  35. [43]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in ICLR, 2019

    2019

  36. [44]

    Tacotron: Towards end-to-end speech synthesis,

    Y . Wang, R. Skerry-Ryan, D. Stanton et al., “Tacotron: Towards end-to-end speech synthesis,” in Interspeech, 2017

    2017

  37. [45]

    Advancing speaker embedding learning: Wespeaker toolkit for research and production,

    S. Wang, Z. Chen, B. Han et al., “Advancing speaker embedding learning: Wespeaker toolkit for research and production,” Speech Communication, vol. 162, p. 103104, 2024

    2024

  38. [46]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for ev- eryone,

    E. Casanova, J. Weber, C. D. Shulby et al. , “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for ev- eryone,” in ICML. PMLR, 2022, pp. 2709–2720

    2022

  39. [47]

    Sdr–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan et al. , “Sdr–half-baked or well done?” in ICASSP. IEEE, 2019, pp. 626–630

    2019

  40. [48]

    A just noticeable dif- ference in c50 for speech,

    J. S. Bradley, R. Reich, and S. Norcross, “A just noticeable dif- ference in c50 for speech,” Applied Acoustics, vol. 58, no. 2, pp. 99–108, 1999

    1999

  41. [49]

    Torchaudio-squim: Reference- less speech quality and intelligibility measures in torchaudio,

    A. Kumar, K. Tan, Z. Ni et al., “Torchaudio-squim: Reference- less speech quality and intelligibility measures in torchaudio,” in ICASSP. IEEE, 2023, pp. 1–5

    2023

  42. [50]

    Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation,

    M. Lavechin, M. M ´etais, H. Titeux et al., “Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation,” ASRU, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.