Pith. sign in

REVIEW 5 major objections 5 minor 29 references

A Multi-Stage Framework for Multimodal Controllable Speech Synthesis

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a three-stage multimodal framework, which aligns face and text encoders to a frozen speech-encoder space and trains the speech synthesizer only on speech, outperforms single-modal face-based and text-prompt-based…

desk verdict A useful three-stage decoupling of multimodal conditioning that deserves refereeing, but the evaluation has enough loose ends that the headline claim isn't fully sealed. read the letter →

arxiv 2506.20945 v1 pith:NWKEF2DD submitted 2025-06-26 cs.SD eess.AS

Rui Niu , Weihao Wu , Jie Chen , Long Ma , Zhiyong Wu This is my paper
classification cs.SDeess.AS
keywords controllablespeechsynthesisface-to-speechtext-promptmultimodalalignmentknowledgedistillationcontrastivelearningspeakerembeddingVITS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the need for fully matched face–speech–prompt triplets with three cheaper paired-data requirements, and claims this makes both face-based and text-prompt-based speech synthesis better than single-modal baselines. In stage one, a face encoder is trained to map faces into the speaker-embedding space of a frozen ECAPA-TDNN speech encoder, using shared-weight classification, knowledge distillation from pretrained face and speech teachers, and contrastive alignment. In stage two, a text encoder is aligned to that same space using both face-caption and speech-caption pairs. In stage three, a VITS model is trained using only high-quality speech as reference input, so synthesis quality does not depend on scarce multimodal data. The reported experiments show lower speaker-verification error rates for face-based synthesis than Face2Speech and Synthesees, higher MOS than FaceTTS and Synthesees, and higher naturalness, similarity, and diversity for text-prompt synthesis than PromptTTS++.

What carries the argument

The central object is a shared speaker-embedding space inherited from the frozen ECAPA-TDNN speech encoder, which acts as the hub for all modalities. Face and text encoders are trained to map into this space by three alignment losses: a shared-weight classification loss based on AMSoftmax with a decision boundary placed between face and speech features, a knowledge-distillation loss that preserves similarity matrices from pretrained face and speech teacher models, and a contrastive alignment loss. The VITS synthesizer is then conditioned on this common space but trained only with speech embeddings. The mechanism works because each alignment stage only needs paired data of one type—face–speech pairs, then face–text and speech–text pairs—so no fully matched triplets are required.

What would settle it

Re-run speaker verification on the LRS3 test set using face-derived speaker embeddings from the trained face encoder: if the equal error rate on unseen speakers rises far above the reported 4.58% and approaches the 11% level of the alignment-only baseline, the staged alignment has not generalized. Equally decisive, a same/different speaker listening test where face-conditioned audio must match the face owner's real voice at better than chance would settle whether the synthesis claim holds.

Watch

Extended reading notes

Core claim

The central discovery is that you do not need fully matched face–speech–text triplets to build a multimodal controllable speech synthesizer: you only need paired data for each alignment step. The face encoder is pushed into the speaker-embedding space of a frozen pretrained speech encoder by combining a shared-weight AMSoftmax classification loss, a joint knowledge-distillation term that transfers similarity matrices from pretrained face and speech teachers, and a contrastive alignment loss; this yields speaker representations from faces that generalize to unseen speakers, with an EER of 4.58% versus 8.23% for Synthesees and 11.05% for Face2Speech. The text encoder is then aligned by the same contrastive loss against both face and speech embeddings, using face captions and speech captions, which widens prompt diversity. Finally, the VITS backbone is trained exclusively on real speech and the frozen speech encoder, so the generative model sees only high-quality speech data; at inference the aligned face or text encoders stand in for the speech encoder. The result, as claimed, is a single model that outperforms single-modal baselines in both face-based and text-prompt-based conditions, with naturalness and similarity scores close to the reference-speech upper bound (N-MOS 3.48, S-MOS 3.63 for faces; N-MOS 3.62, S-MOS 3.75 for prompts).

Load-bearing premise

The load-bearing premise is that face and text embeddings aligned by the three losses land close enough to the real speech-encoder distribution that VITS, trained only on speech embeddings, keeps producing natural, speaker-consistent audio when it conditions on those other modalities.

Editorial extensions

If this is right

  • A single trained model can generate speech from speech, face, or text-prompt references, so an application does not need to know the input modality in advance.
  • The speech-quality ceiling depends on the quantity and quality of the speech corpus used in the final training stage, not on the size of matched multimodal corpora.
  • New caption sources from either faces or speech can broaden prompt diversity without retraining the synthesizer.
  • Because alignment is separated from generation, the same face and text encoders can be plugged into other end-to-end TTS backbones besides VITS.
  • Face-conditioned synthesis gains generalization from pretrained face recognition and speaker verification teachers, reducing the amount of face–speech paired data needed for robust speaker identity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not say this, but the whole chain is capped by the frozen speech encoder: if the speaker space it defines is biased or coarse, face and text alignment cannot add information the encoder does not already carry, so a stronger speech encoder would likely improve every modality at once.
  • The paper reports silhouette score as evidence of diversity; a direct test would be semantic clustering of generated utterances over novel prompt combinations not present in the caption datasets, which the paper does not report.
  • The same staged recipe could apply to other conditional generators: train a strong embedding-conditioned generator on abundant data from one modality, then align sparse auxiliary modalities to that embedding space with the three losses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents a three-stage training framework for multimodal controllable speech synthesis. In Stage 1, a face encoder is trained to align with a frozen ECAPA-TDNN speech encoder using a shared-weight AMSoftmax classification loss, knowledge distillation from pretrained speech and face teachers, and contrastive learning. In Stage 2, a text encoder is trained on face-text and speech-text paired data to map text prompts into the same speaker-embedding space. In Stage 3, a VITS-based speech synthesis model is trained exclusively on speech-encoder embeddings, with the intention that at inference the aligned face or text encoders can supply the conditioning. Experiments report objective speaker-verification metrics for the face modality, MOS for face-based and text-prompt-based synthesis, and a silhouette-score-based diversity measure. The paper claims that the proposed method outperforms single-modal baselines (FaceTTS, Synthesees, Face2Speech, PromptTTS++) for both face-based and text-prompt-based synthesis.

Significance. If the reported results hold, the framework is a meaningful practical contribution: it avoids the need for fully matched speech-face-text triplets, instead requiring only three types of paired data, and it offers a unified way to condition TTS on either a face image or a text prompt. The staged training design is sensible and the use of a frozen speech encoder as an anchor is a reasonable way to transfer speaker information across modalities. However, the evaluation has several load-bearing weaknesses that currently temper the strength of the central claim. These include an unexplained violation of the stated reference-speech upper bound on speaker-consistency MOS, the absence of any direct check that face/text embeddings lie in the VITS conditioning manifold, and a nonstandard interpretation of the silhouette score. The paper does not make code available, but it does provide a demo URL, and the objective EER/minDCF results, if reproducible, indicate that the face encoder produces discriminative speaker representations.

major comments (5)
  1. [Table III, Section IV-C] The face-based speaker-consistency MOS for the proposed method (3.63 ± 0.10) exceeds the Reference Speech condition (3.58 ± 0.10), which is explicitly described as an upper bound for face-based synthesis. This internal inconsistency is not addressed anywhere in the paper. It suggests that the subjective evaluation may be noisy, biased, or confounded (e.g., different reference utterances, listeners not blinded to condition, or different numbers of ratings). Because the central claim of outperforming face-based baselines rests on these MOS numbers, the authors must either explain why the face condition can legitimately exceed the speech-reference condition, or re-run the listening test with a protocol that makes the upper-bound relationship meaningful.
  2. [Sections III-C and IV-C] The face and text encoders are trained only with alignment losses to mimic a frozen ECAPA-TDNN speech encoder, while the VITS backbone is trained exclusively on ECAPA embeddings. The paper does not provide any distribution-level analysis (e.g., embedding distance statistics, density estimates, or fine-tuning/adaptation experiments) to show that the face and text embeddings fall within the support of the ECAPA embedding distribution that VITS conditions on. The reported EER/minDCF results only demonstrate that the face encoder produces discriminative embeddings in the ECAPA space, not that those embeddings are suitable as VITS conditioning inputs. Although the MOS results provide some end-to-end evidence, the S-MOS anomaly in Table III weakens that evidence. A direct analysis, such as comparing embedding-space distances for matched speakers or measuring VITS output when conditioning on face/text versus speech embeddings, is needed to support the central claim.
  3. [Section IV-D, Table IV] The silhouette score is interpreted as "a lower contour coefficient indicates a larger distribution area for the cluster, thus demonstrating superior diversity." This is a nonstandard reading of the silhouette score, which is a clustering-quality measure: a lower value can also indicate poor cluster separation or noisy embeddings. Since this metric is the only objective evidence for the diversity claim in text-prompt synthesis, the authors should justify this interpretation, report silhouette scores for appropriate reference conditions (e.g., ground-truth speech or reference-speech synthesis), and ideally supplement it with additional diversity metrics (e.g., objective speaker-embedding variance, human judgments of gender consistency).
  4. [Section IV-A] The paper states that a "large language model" is used to extract keywords from captions, but the specific model (e.g., GPT-4, Llama-3, or a local model) and the prompting procedure are not given. Because the constructed prompts are the training input for the text encoder, this missing detail makes the text-prompt pipeline irreproducible and could materially affect the reported text-prompt synthesis results. Please name the LLM and provide the extraction algorithm or examples, or at least an ablation showing the sensitivity of the results to this choice.
  5. [Tables I and II, Section IV-B] The objective EER and minDCF results are reported as single values without confidence intervals or significance tests. The claim that the proposed method "significantly outperforms" baselines is not supported by any statistical analysis. Since these numbers are a central part of the face-based evaluation, please provide multiple training runs or bootstrap confidence intervals, and perform a significance test (e.g., paired bootstrap or Wilcoxon) where appropriate.
minor comments (5)
  1. [Section III-B] The text says the text encoder is trained using "the previously described alignment loss," but Eq. (8) only shows the contrastive loss. It is unclear whether the classification loss L_ce and knowledge-distillation loss L_kd are also applied to the text encoder, and if so, how the shared weight W or the teacher similarity matrices are adapted for text. Please clarify the exact loss composition for Stage 2.
  2. [Section IV-B] The phrase "A lower contour coefficient indicates a larger distribution area for the cluster" is confusing; the standard term is "silhouette score." Please define the metric formally and state why a lower value is desirable in this context.
  3. [Section IV-C] The sentence "This result proved that the three-stage training strategy with high-quality training data can produce more natural speech" overstates what a single MOS comparison can prove. The paper does not include an ablation that varies training-data quality, so the phrase "proved" should be softened.
  4. [Section I and IV-A] There is a minor typo: "We compared it with several baseline models" should be "We compared it with several baseline models." Also, the demo URL in the footnote is not referenced in the conclusion; please ensure the demo link is consistently cited.
  5. [References] Reference [15] is about object detection and does not directly support the claim of "mode collapse" in speaker-embedding alignment. A more relevant reference on representation collapse in metric learning or multimodal alignment would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the multi-stage alignment pipeline is supported by held-out evaluation and independent synthesis metrics.

full rationale

The paper's derivation chain is not circular. Stage 1 trains a face encoder with classification, knowledge distillation, and contrastive losses (Eqs. 1-7) to align with a frozen ECAPA-TDNN speech encoder. Stage 2 trains a text encoder with contrastive alignment to the face and speech embeddings (Eq. 8). Stage 3 trains VITS using only speech-encoder embeddings, then uses the aligned encoders at inference. The objective EER/minDCF metrics are computed in the same ECAPA embedding space that the encoders are trained to mimic, so those numbers are partly an expected consequence of the training objective. However, the evaluation uses the held-out LRS3 test speakers described in Section IV-A, and the face-based claim is also supported by human N-MOS and S-MOS on synthesized audio, which are independent of the alignment losses. The text-prompt claim is supported by N-MOS, S-MOS, and silhouette scores on generated samples, none of which reduce by construction to the training losses. The absence of explicit distribution matching between face/text embeddings and the VITS conditioning manifold is a real robustness risk, but it is a generalization concern rather than a circular step. No load-bearing self-citation, authored uniqueness theorem, or ansatz-smuggling citation appears in the argument. The anomalous S-MOS result where face-based Ours (3.63) exceeds reference speech (3.58) suggests subjective evaluation noise, but it does not make the central claim circular. Overall, the core synthesis-quality result rests on held-out and human evaluations that are not defined in terms of the fitted encoders.

Assumptions & free parameters 11 free parameters · 5 assumptions · 0 invented entities

The framework rests on several domain assumptions about cross-modal predictiveness and on eleven manually chosen hyperparameters. No new physical entities are introduced. The most consequential assumptions are that face appearance and facial-attribute text carry voice-relevant information and that the fixed ECAPA-TDNN embedding space is a universal conditioning space for VITS.

free parameters (11)
  • alpha (speech classification loss weight) = 0.1
    Balances the two modality terms in Eq. (2).
  • m (AMSoftmax margin) = 0.2
    Margin in the shared-weight classification loss, Eq. (1).
  • s (AMSoftmax scale) = 30
    Scale in Eq. (1).
  • mu (teacher similarity balance) = 0.8
    Weights speech vs face teacher similarity matrices in Eq. (4).
  • beta (same-modality distillation weight) = 0.1
    Weights the within-modality term in Eq. (5).
  • tau (contrastive temperature) = 0.1
    Temperature in Eqs. (6) and (8).
  • gamma (overall distillation weight) = 10
    Balances KD loss in Eq. (7).
  • Stage 1 optimization schedule = batch 96, lr 0.0002, 200,000 steps
    Hyperparameters for the face encoder; no sensitivity analysis reported.
  • Stage 2 optimization schedule = batch 64+64, lr 0.0002, 200,000 steps
    Hyperparameters for the text encoder; no sensitivity analysis reported.
  • Stage 3 optimization schedule = batch 96, lr 0.0002, 600,000 steps
    Hyperparameters for VITS; no sensitivity analysis reported.
  • Unspecified LLM for keyword extraction = not reported
    Face captions from CelebAText-HQ are converted to prompts using an unnamed large language model; this affects all text-prompt results and is not reproducible.
assumptions (5)
  • domain assumption A frozen ECAPA-TDNN speaker embedding space, pretrained on VoxCeleb2, is a suitable conditioning space for controlling timbre, prosody, and style in VITS-based TTS.
    Invoked in Sections III-A and III-C; the entire framework trains other encoders to mimic this space and VITS is conditioned on it.
  • domain assumption Static facial appearance contains sufficient information about a person's voice characteristics to predict a useful speaker embedding.
    Basis of the face-based branch; training on LRS3 face-speech pairs assumes this mapping is learnable and generalizes to unseen faces.
  • domain assumption Text captions describing facial attributes (e.g., 'square face, small ears') and speaking style can be mapped to the same speaker embedding space and increase diversity.
    Used in Section III-B; CelebAText-HQ captions are keyword-extracted by an unspecified LLM and treated as text prompts.
  • domain assumption Knowledge distillation from a face-recognition teacher (FaceNet) and a speech teacher preserves discriminative structure and prevents mode collapse after modality alignment.
    Section III-A.2; the distillation loss relies on teacher embedding quality and on the union of similarity matrices being a good target.
  • standard math The loss formulation and optimization procedures (AMSoftmax, contrastive learning, Adam) are standard and stable for the stated batch sizes and step counts.
    Eqs. (1)-(8) use standard losses with no novel theory; convergence is assumed from hyperparameter choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multi-Stage Framework for Multimodal Controllable Speech Synthesis." pith.science (2026). https://pith.science/paper/NWKEF2DD

@misc{pith2026250620945,
  author       = {Pith},
  title        = {Pith review of: A Multi-Stage Framework for Multimodal Controllable Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWKEF2DD}},
  note         = {Machine review of arXiv:2506.20945}
}
read the original abstract

Controllable speech synthesis aims to control the style of generated speech using reference input, which can be of various modalities. Existing face-based methods struggle with robustness and generalization due to data quality constraints, while text prompt methods offer limited diversity and fine-grained control. Although multimodal approaches aim to integrate various modalities, their reliance on fully matched training data significantly constrains their performance and applicability. This paper proposes a 3-stage multimodal controllable speech synthesis framework to address these challenges. For face encoder, we use supervised learning and knowledge distillation to tackle generalization issues. Furthermore, the text encoder is trained on both text-face and text-speech data to enhance the diversity of the generated speech. Experimental results demonstrate that this method outperforms single-modal baseline methods in both face based and text prompt based speech synthesis, highlighting its effectiveness in generating high-quality speech.

Figures

Figures reproduced from arXiv: 2506.20945 by the authors.

Figure 1
Figure 1. A unified multimodal controllable speech synthesis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the training process for the Face Encoder (left) and the Text Encoder (right). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training process and inference process for VITS based speech synthesis model. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages

  1. [1]

    Mega- tts 2: Boosting prompting mechanisms for zero-shot speech synthesis,

    Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, et al., “Mega- tts 2: Boosting prompting mechanisms for zero-shot speech synthesis,” inThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  2. [2]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al., “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    arXiv 2024

  3. [3]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer,

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu, “Maskgct: Zero-shot text-to-speech with masked generative codec transformer,”arXiv preprint arXiv:2409.00750, 2024

    arXiv 2024

  4. [4]

    Imaginary voice: Face-styled diffusion model for text-to-speech,

    Jiyoung Lee, Joon Son Chung, and Soo-Whan Chung, “Imaginary voice: Face-styled diffusion model for text-to-speech,” inICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    work page 2023

  5. [5]

    SYNTHE-SEES: Face based text-to-speech for virtual speaker,

    Jae Hyun Park, Joon-Gyu Maeng, TaeJun Bak, and Young-Sun Joo, “SYNTHE-SEES: Face based text-to-speech for virtual speaker,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 10321–10325

    work page 2024

  6. [6]

    Face2Speech: Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image.,

    Shunsuke Goto, Kotaro Onishi, Yuki Saito, Kentaro Tachibana, and Koichiro Mori, “Face2Speech: Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image.,” in INTERSPEECH, 2020, pp. 1321–1325

    work page 2020

  7. [7]

    FVTTS : Face based voice synthesis for text-to-speech,

    Minyoung Lee, Eunil Park, and Sungeun Hong, “FVTTS : Face based voice synthesis for text-to-speech,” inInterspeech 2024, 2024, pp. 4953– 4957

    work page 2024

  8. [8]

    Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt,

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng, “Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    work page 2024

Show all 29 references
  1. [9]

    Prompttts: Controllable text-to-speech with text descriptions,

    Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan, “Prompttts: Controllable text-to-speech with text descriptions,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2023, pp. 1–5, IEEE

    2023

  2. [10]

    PromptTTS++: Controlling speaker identity in prompt-based text-to- speech using natural language descriptions,

    Reo Shimizu, Ryuichi Yamamoto, Masaya Kawamura, Yuma Shi- rahata, Hironori Doi, Tatsuya Komatsu, and Kentaro Tachibana, “PromptTTS++: Controlling speaker identity in prompt-based text-to- speech using natural language descriptions,” inICASSP 2024-2024 IEEE International Confer...

    2024

  3. [11]

    Mm-tts: A unified framework for multimodal, prompt-induced emotional text-to-speech synthesis,

    Xiang Li, Zhi-Qi Cheng, Jun-Yan He, Xiaojiang Peng, and Alexan- der G Hauptmann, “Mm-tts: A unified framework for multimodal, prompt-induced emotional text-to-speech synthesis,”arXiv preprint arXiv:2404.18398, 2024

    2024 arXiv

  4. [12]

    MM-TTS: Multi- modal prompt based style transfer for expressive text-to-speech synthe- sis,

    Wenhao Guan, Yishuang Li, Tao Li, Hukai Huang, Feng Wang, Jiayan Lin, Lingyan Huang, Lin Li, and Qingyang Hong, “MM-TTS: Multi- modal prompt based style transfer for expressive text-to-speech synthe- sis,” inProceedings of the AAAI Conference on Artificial Intelligence, 2024, ...

    2024

  5. [13]

    Gen- eralized end-to-end loss for speaker verification,

    Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno, “Gen- eralized end-to-end loss for speaker verification,” in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4879–4883

    2018

  6. [14]

    Additive margin softmax for face verification,

    Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu, “Additive margin softmax for face verification,”IEEE Signal Processing Letters, vol. 25, no. 7, pp. 926–930, 2018

    2018

  7. [15]

    Bridging the gap between object and image-level representations for open-vocabulary detection,

    Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan, “Bridging the gap between object and image-level representations for open-vocabulary detection,” Advances in Neural Information Processing Systems, vol. 35, pp. 33781– 33794, 2022

    2022

  8. [16]

    Joint- teaching: Learning to refine knowledge for resource-constrained un- supervised cross-modal retrieval,

    Peng-Fei Zhang, Jiasheng Duan, Zi Huang, and Hongzhi Yin, “Joint- teaching: Learning to refine knowledge for resource-constrained un- supervised cross-modal retrieval,” inProceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 1517–1525

    2021

  9. [17]

    Represen- tation learning with contrastive predictive coding,

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals, “Represen- tation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  10. [18]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    Jaehyeon Kim, Jungil Kong, and Juhee Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in International Conference on Machine Learning. 2021, pp. 5530–5540, PMLR

    2021

  11. [19]

    Lrs3- ted: a large-scale dataset for visual speech recognition,

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman, “Lrs3- ted: a large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  12. [20]

    Multi- caption text-to-face synthesis: Dataset and algorithm,

    Jianxin Sun, Qi Li, Weining Wang, Jian Zhao, and Zhenan Sun, “Multi- caption text-to-face synthesis: Dataset and algorithm,” inProceedings of the 29th ACM International Conference on Multimedia, New York, NY , USA, 2021, Mm ’21, pp. 2290–2298, Association for Computing Machinery

    2021

  13. [21]

    LibriTTS-p: A corpus with speaking style and speaker identity prompts for text-to-speech and style caption- ing,

    Masaya Kawamura, Ryuichi Yamamoto, Yuma Shirahata, Takuya Ha- sumi, and Kentaro Tachibana, “LibriTTS-p: A corpus with speaking style and speaker identity prompts for text-to-speech and style caption- ing,”arXiv preprint arXiv:2406.07969, 2024

    2024 arXiv

  14. [22]

    Libritts-r: A restored multi-speaker text-to-speech corpus,

    Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna, “Libritts-r: A restored multi-speaker text-to-speech corpus,” arXiv preprint arXiv:2305.18802, 2023

    2023 arXiv

  15. [23]

    Libritts: A corpus derived from librispeech for text-to-speech,

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu, “Libritts: A corpus derived from librispeech for text-to-speech,”arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  16. [24]

    Facenet: A unified embedding for face recognition and clustering,

    Florian Schroff, Dmitry Kalenichenko, and James Philbin, “Facenet: A unified embedding for face recognition and clustering,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815–823

    2015

  17. [25]

    Vggface2: A dataset for recognising faces across pose and age,

    Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zis- serman, “Vggface2: A dataset for recognising faces across pose and age,” in2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, 2018, pp. 67–74

    2018

  18. [26]

    Joint face detection and alignment using multitask cascaded convolutional networks,

    Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,”IEEE signal processing letters, vol. 23, no. 10, pp. 1499– 1503, 2016

    2016

  19. [27]

    Ecapa- tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck, “Ecapa- tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”arXiv preprint arXiv:2005.07143, 2020

    2005 arXiv

  20. [28]

    V oxceleb2: Deep speaker recognition,

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman, “V oxceleb2: Deep speaker recognition,”arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  21. [29]

    Explor- ing the limits of transfer learning with a unified text-to-text transformer,

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Explor- ing the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.