Pith. sign in

REVIEW 4 major objections 5 minor 40 references

A lightweight adapter can turn a static face photo into natural speech by mapping it into the style space of a frozen text-to-speech model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 22:24 UTC pith:LYYTNG6J

load-bearing objection Solid Freeze-Align engineering on StyleTTS 2: real but modest face→style signal, honest about limits, evaluation still automatic-only and thin. the 4 major comments →

arxiv 2607.26742 v1 pith:LYYTNG6J submitted 2026-07-29 eess.AS cs.AIcs.SD

Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model

Carlos Mu\~noz-Romero , Jose A. Gonzalez-Lopez This is my paper
classification eess.AS cs.AIcs.SD
keywords Face-to-Speechzero-shot TTSlatent space alignmentstyle diffusion modelscross-lingual transferFace AdapterStyleTTS 2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard voice cloning needs a short audio sample, which blocks synthesis for anyone known only by image—historical figures, game characters, or silent video. This paper shows that a small Face Adapter, plus light tuning of a face network’s upper layers, can project a face-recognition embedding into the style space of a frozen StyleTTS 2 model. On speakers never seen in training, the resulting speech scores as natural as or more natural than real recordings by automatic metrics, ranks the correct speaker above chance in face-to-voice retrieval, and stays consistent with the target identity. The same English-trained adapter produces fluent Spanish without any retraining, indicating the face-to-style link is largely language-independent. The result is usable voice from appearance alone while leaving a high-quality acoustic generator untouched.

Core claim

A Freeze-Align Face-to-Speech system synthesizes natural, identity-consistent speech from one static face by aligning face-recognition features to the style space of a frozen StyleTTS 2 model via a trainable Face Adapter and soft-tuning of the face encoder’s upper blocks. On held-out LRS3 identities, automatic naturalness matches or exceeds ground truth, face-to-voice retrieval stays above chance, and an English-trained adapter transfers zero-shot to fluent Spanish.

What carries the argument

The Freeze-Align Face Adapter: a small MLP that maps 512-d face embeddings into the 128-d style space of a frozen StyleTTS 2 teacher, trained with a hybrid contrastive, relational-distillation, variance, and demographic loss while only the adapter and upper face-encoder blocks are updated.

Load-bearing premise

Automatic naturalness scores and embedding similarities on only twenty-four held-out speakers, without a full human listening test, are enough to prove the voice truly matches the person and sounds as good as real speech.

What would settle it

A blinded human MOS and speaker-similarity test on the same unseen identities in which listeners rate the speech well below real recordings or cannot match faces to voices above chance would overturn the claim of perceptual naturalness and speaker consistency.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Voices can be synthesized for people or characters known only from images, with no audio reference required.
  • High-quality TTS backbones need not be retrained end-to-end to support face-conditioned synthesis.
  • A face-to-style mapping trained in one language can drive speech in another once the acoustic backbone exists.
  • Inference-time blending of face-derived timbre with diffusion prosody supplies an explicit identity-versus-naturalness control.
  • Soft-tuning only the upper face-encoder blocks is enough to raise absolute style alignment without full backbone fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The modest face contribution to absolute identity (SECSaudio roughly 0.56–0.64) implies most speaker character still comes from the TTS prior, so strong biometric match may still need audio or richer visual–vocal models.
  • If the language-agnostic mapping generalizes, one face adapter could serve many languages once each acoustic backbone is available.
  • The same freeze-and-align pattern could be tried on other frozen generators whose interface is a style or speaker embedding.
  • Faces far outside the TED-talk demographic remain an open test of whether the weak biometric signal holds beyond the training domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Freeze-Align Face-to-Speech system that maps a static face image into the 128-d style space of a frozen StyleTTS 2 model via a lightweight MLP Face Adapter and soft-tuning of the upper blocks of an InceptionResnetV1 face encoder. Training uses a hybrid loss (multi-positive InfoNCE, relational knowledge distillation, variance regularization, and demographic auxiliary heads) on LRS3 face–audio pairs. At inference, timbre and prosody are decoupled with blend weights α and β. On 24 held-out LRS3 identities the authors report above-chance face→voice retrieval, SECS_emb ≈ 0.40–0.42, SECS_audio ≈ 0.61, UTMOS 3.7–4.0 (vs. GT 3.61), no mode collapse (SED), a soft-tuning ablation (0.29→0.42 SECS_emb), an α identity–naturalness trade-off, and zero-shot transfer of the English-trained adapter to a Spanish-fine-tuned backbone.

Significance. If the claims hold, the work offers a practical and cleanly engineered route to voice generation when only a face is available (historical figures, game NPCs), without retraining a large TTS generator. Keeping StyleTTS 2 frozen, exposing an explicit α control, documenting a soft-tuning ablation, and demonstrating language-agnostic transfer are genuine strengths. The contribution is incremental rather than foundational: the face’s net identity contribution is modest relative to the TTS prior, and the evaluation does not yet securely establish perceptual speaker–face consistency. Still, the Freeze-Align framing and hybrid loss design are useful for the F2S community and are worth publishing once evaluation gaps are closed.

major comments (4)
  1. [Section 4.4, Table 5] Section 4.4 and Table 5 show SECS_audio only ranges from 0.56 (α=0.9, diffusion-dominant) to 0.64 (α=0.3). The authors correctly note that most absolute identity comes from the StyleTTS 2 prior. The central claim that the generated voice is “consistent with the target speaker” therefore rests on a modest delta. Without a mismatched-face (or shuffled-identity) control that holds the prior fixed, it is not possible to isolate how much of the above-chance retrieval or SECS_audio is truly face-driven versus prior-driven. This control is load-bearing and should be added.
  2. [Section 3.3, Table 3] Evaluation uses only N=24 unseen speakers (Section 3.3, Table 3). Top-1 is noisy; McNemar tests find no significant differences among loss configurations (all p>0.1). Above-chance Top-1/Top-5 is encouraging but, at this N, does not securely support configuration comparisons or a strong biometric claim. Either enlarge the held-out identity pool or report confidence intervals / bootstrap tests and temper claims that depend on ranking precision.
  3. [Abstract; Sections 3.3–4.3; Table 4] The abstract and Section 4.3 assert high naturalness matching ground truth and voice consistency with the target speaker, yet there is no human MOS or AB listening test. UTMOS/NISQA are automatic English-biased proxies; checkpoint selection used the same unseen development split (declared mild optimism). For a perceptual synthesis claim, at least a small formal listening study (naturalness + face–voice match / AB preference) is needed before the strongest wording is justified.
  4. [Section 4.1] Section 4.1 compares SECS_audio (0.61 EN) to prior end-to-end F2S numbers (≈0.75) while noting the protocols are not matched and leaving a StyleTTS 2 end-to-end baseline as future work. That baseline is important for positioning: without it, readers cannot tell whether Freeze-Align underperforms because of the frozen teacher or because the face signal is weak. A protocol-matched comparison, or a clearer quantitative bound on the face contribution, should be provided or the comparison reframed more carefully.
minor comments (5)
  1. [Figure 1, Section 2] Figure 1 caption and body use z_f ace / h_vis with inconsistent spacing/subscripts; unify notation for z_face, z_audio, h_vis throughout.
  2. [Table 3] Table 3 states EN and ES embedding metrics are not cross-comparable, yet Spanish Top-1/Top-5 look higher; a brief note on why retrieval rates differ (style-space geometry, speaker count in the Spanish backbone) would help.
  3. [Section 2.3, Eq. (6)] Eq. (6): clarify the default when no reference audio is available (z_ref ← z_face) earlier in the inference subsection; it is easy to miss.
  4. [Section 1] Typos / formatting: “ZipV oice”, “V oxCPM”, “VibeV oice”, “mode collapse[18]” (missing space), and “androgens jointly drive…” citation clustering could be cleaned for camera-ready.
  5. [Footnote 1] Code and audio samples are promised “upon acceptance”; providing an anonymous supplementary link during review would strengthen reproducibility claims.

Circularity Check

0 steps flagged

No significant circularity: empirical face–style alignment evaluated on held-out identities with external metrics.

full rationale

This is a standard multimodal adaptation paper. The Face Adapter is trained with InfoNCE/RKD/variance/auxiliary losses to map face embeddings into a frozen StyleTTS 2 style space; claims of naturalness, above-chance retrieval, speaker consistency, and cross-lingual transfer are then checked on 24 identities never seen in training, using UTMOS/NISQA, external Resemblyzer SECS_audio, and ranking against real-audio style prototypes. None of these quantities is algebraically identical to a fitted training input: SECS_emb is related to the contrastive objective but is measured on held-out speakers (generalization, not definition), SECS_audio uses an independent speaker encoder, and UTMOS is an external MOS proxy. The α-decoupling analysis further shows the face contribution is modest rather than forced. Citations (StyleTTS 2, FaceNet/VGGFace2, RKD, CLIP-style InfoNCE, prior F2S) are to external work, not load-bearing self-theorems. Checkpoint selection on the unseen development split is mild evaluation optimism, not a circular derivation. No self-definitional step, fitted-as-prediction step, or uniqueness import is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The claim rests on standard deep-learning practice plus domain assumptions about face–voice correlation and the suitability of StyleTTS 2’s style vector as the alignment interface. Free parameters are the usual loss weights, temperature, α/β, and architecture sizes chosen or tuned on data. No new physical entities; the Face Adapter is an engineered module, not a postulated natural object.

free parameters (4)
  • λ_nce, τ, λ_rkd, λ_gen, λ_age, λ_var (loss configuration) = e.g. Balanced: λ_nce=1.0, τ=0.07, λ_gen=0.2, λ_age=0.1, λ_rkd=0.5; Identity-priority τ=0.04, λ_nce=2.0
    Hand-chosen / configuration-swept weights and CLIP-style temperature that define the four operating points in Table 2; central retrieval and SECS numbers depend on these choices.
  • inference α (timbre blend) and β (prosody blend) = default α=0.5, β=0.9
    User-facing knobs in Eq. (6); reported naturalness and SECS_audio are quoted at α=0.5, β=0.9 and the α sweep shows large UTMOS movement, so headline quality depends on this choice.
  • Face Adapter MLP width/depth and soft-tune block selection = ~20M trainable params; lower face layers frozen
    512→1024→1024→128 MLP (~1.7M) and which upper InceptionResnetV1 blocks are trainable (~18.3M) are design choices that determine alignment capacity versus forgetting.
  • AdamW lr, batch construction K=4, 200 epochs, checkpoint selection criterion = lr 2e-3, batch 1024 via balanced sampler, 200 epochs
    Training hyperparameters and the composite identity/naturalness/collapse criterion used to pick the reported checkpoint on a disjoint dev set.
axioms (5)
  • domain assumption Facial appearance and vocal identity share a weak but usable statistical correlation (shared developmental morphology).
    Stated in Introduction with citations to matching studies; without this, face→style alignment cannot beat chance on unseen identities.
  • domain assumption StyleTTS 2’s 128-d style vector factors timbre vs prosody enough that a static face can supply timbre while diffusion/reference supplies prosody.
    Section 2.1 and 2.3; the whole Freeze-Align interface and Eq. (6) decoupling rest on this factorization.
  • ad hoc to paper Contrastive multi-positive InfoNCE + RKD + variance + demographic aux heads prevent mode collapse better than MSE regression in weakly correlated face–voice spaces.
    Section 2.2 motivates the hybrid loss; configurations are compared but not derived from a theorem.
  • domain assumption Automatic UTMOS/NISQA and Resemblyzer SECS are adequate proxies for naturalness and speaker identity in lieu of large-scale human listening tests.
    Section 4.3 explicitly substitutes proxies; Spanish results are labeled indicative because proxies are English-biased.
  • standard math Standard supervised learning, cosine similarity retrieval, and McNemar testing are valid evaluation tools on N=24 unseen speakers.
    Section 3.3 evaluation protocol.
invented entities (2)
  • Face Adapter (MLP projecting h_vis to z_face in StyleTTS 2 style space) no independent evidence
    purpose: Trainable map from face-recognition embedding to frozen TTS style vector under the Freeze-Align recipe.
    Standard adapter module, not a new natural kind; listed for completeness as the paper’s main engineered object.
  • Freeze-Align F2S framework no independent evidence
    purpose: Name for keeping StyleTTS 2 frozen while aligning face features via adapter + soft-tuning.
    Methodological label rather than ontological invention; independent evidence would be external replications, which are not yet available.

pith-pipeline@v1.2.0-daily-grok45 · 14669 in / 3999 out tokens · 85815 ms · 2026-07-30T22:24:10.811656+00:00 · methodology

0 comments
read the original abstract

Zero-shot text-to-speech (TTS) clones a voice from a short audio prompt, but this reliance on reference audio is a barrier when only visual information is available, e.g. for historical figures or video-game characters. In this work, we propose a Face-to-Speech (F2S) framework that predicts a plausible voice from a static facial image. A lightweight Face Adapter, together with soft-tuning of the face encoder's upper blocks, aligns face-recognition features with the style space of a frozen StyleTTS 2 model, kept frozen during training. We evaluate on held-out identities from LRS3, a large-scale audiovisual corpus of English TED-talk videos. The synthesized speech is highly natural (UTMOS 3.7-4.0, matching or exceeding the 3.61 of ground truth), face-to-voice retrieval is consistently above chance, and the generated voice is consistent with the target speaker. Without any retraining, an English-trained adapter also produces fluent Spanish speech, indicating that the face-to-style mapping is largely language-agnostic.

Figures

Figures reproduced from arXiv: 2607.26742 by Carlos Mu\~noz-Romero, Jose A. Gonzalez-Lopez.

Figure 1
Figure 1. Figure 1: Proposed Freeze-Align F2S architecture used during training. An InceptionResnetV1 backbone maps the input face to a 512-d embedding hvis, which the trainable Face Adapter (MLP, 512 → 1024 → 1024 → 128) projects into the 128-d style space zface. The frozen StyleTTS 2 style encoder provides the acoustic target zaudio. Training aligns the two modalities through the hybrid loss (InfoNCE, RKD, variance, and dem… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 8 linked inside Pith

  1. [1]

    Introduction Recent advances in zero-shot text-to-speech (TTS) enable voice cloning from a few seconds of reference audio [1, 2, 3, 4]. Systems such as Tacotron 2 [5], FastSpeech 2 [6] and HiFi- GAN [7], along with recent large-corpus models (ZipV oice [2], V oxCPM [3], VibeV oice [4]), clone an arbitrary voice from a short audio prompt. StyleTTS 2 [8], i...

  2. [2]

    Architectural overview The overall architecture (Fig

    Proposed Method 2.1. Architectural overview The overall architecture (Fig. 1) comprises a visual backbone, a trainable projection module (Face Adapter), and a frozen acoustic generator (StyleTTS 2 [8]). We choose StyleTTS 2 because its style-diffusion mechanism condenses timbre and 1Links to the code repository and audio samples will be provided upon pape...

    Pith/arXiv arXiv 2026

  3. [3]

    Datasets and preprocessing We evaluate our proposed method in both English and Span- ish

    Experimental Setup 3.1. Datasets and preprocessing We evaluate our proposed method in both English and Span- ish. The Face Adapter is trained and evaluated on LRS3 [13], a dataset of English TED-talk videos. Starting from the vali- dated record list of prior F2S work and applying quality filtering (low-quality audio, undetectable faces, audiovisual desync...

  4. [4]

    Results and Discussion 4.1. Comparison with prior F2S work For context, prior end-to-end F2S systems report SECS on syn- thesized audio of0.754(FVTTS [9]) and0.748(Face-TTS [14]) on LRS3, while our SECS audio is0.61(EN) and0.57 (ES, cross-lingual). These arenot directly comparable: prior systems fine-tune the full TTS end-to-end, whereas we keep the acous...

  5. [5]

    The mapping holds for unseen identities, with natu- ralness on par with ground truth (UTMOS3.7–4.0vs.3.61), above-chance retrieval, and zero-shot transfer to fluent Spanish

    Conclusions We presented a Freeze-Align framework that synthesizes a plau- sible, coherent voice from a single face by adapting a face- recognition embedding to the style space of a frozen StyleTTS 2 teacher, via a trainable Face Adapter and soft-tuning of the face encoder. The mapping holds for unseen identities, with natu- ralness on par with ground tru...

  6. [6]

    Acknowledgments This work was supported by Monoceros Labs; the R&D&I project C-HUM-223-UGR23, co-financed by the Consejer ´ıa de Universidad, Investigaci ´on e Innovaci ´on and the Euro- pean Union through the FEDER Andalusia 2021–2027 Pro- gramme; and grants PID2022-141378OB-C22 and AIA2025- 163317-C32, funded by MICIU/AEI/10.13039/501100011033 and ERDF/EU

    doi:10.13039/501100011033 2021

  7. [7]

    Generative AI Use Disclosure The authors used generative AI tools for language editing and coding assistance only, not to generate scientific content, design the methodology, or interpret the results; the authors reviewed all content and take full responsibility for the publication

  8. [8]

    StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,

    Y . A. Li, C. Han, V . S. Raghavan, G. Mischler, and N. Mes- garani, “StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 19 594–19 621

    2023

  9. [9]

    FVTTS: Face based voice syn- thesis for text-to-speech,

    M. Lee, E. Park, and S. Hong, “FVTTS: Face based voice syn- thesis for text-to-speech,” inProc. Interspeech 2024, 2024, pp. 4953–4957

    2024

  10. [10]

    A survey on neural speech synthesis,

    X. Tan, T. Qin, F. Soong, and T.-Y . Liu, “A survey on neural speech synthesis,”arXiv preprint arXiv:2106.15561, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    ZipV oice: Fast and high-quality zero-shot text-to-speech with flow matching,

    H. Zhu, W. Kang, Z. Yao, L. Guo, F. Kuang, Z. Li, W. Zhuang, L. Lin, and D. Povey, “ZipV oice: Fast and high-quality zero-shot text-to-speech with flow matching,”arXiv preprint arXiv:2506.13053, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    V oxCPM: Tokenizer-free TTS for context-aware speech generation and true- to-life voice cloning,

    Y . Zhou, G. Zeng, X. Liu, X. Li, R. Yu, Z. Wanget al., “V oxCPM: Tokenizer-free TTS for context-aware speech generation and true- to-life voice cloning,”arXiv preprint arXiv:2509.24650, 2025

    arXiv 2025

  13. [13]

    VibeV oice technical report,

    Z. Peng, J. Yu, W. Wang, Y . Chang, Y . Sun, L. Dong, Y . Zhu, W. Zhou, J. Bao, Z. Wanget al., “VibeV oice technical report,” arXiv preprint arXiv:2508.19205, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerry-Ryan, R. A. Saurous, Y . Agiomyrgiannakis, and Y . Wu, “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2018, pp. 4779–4783

    2018

  15. [15]

    Freeze-Align

    or emotion [16] on corpora such as V oxCeleb2 [17]. De- spite this potential, F2S faces a critical challenge: the weak, non-deterministic correlation between facial and vocal traits. Approaches trained with simple regression objectives tend to suffer frommode collapse[18], where generated voices con- verge toward generic, gender-averaged prototypes [12]. ...

  16. [16]

    FastSpeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “FastSpeech 2: Fast and high-quality end-to-end text to speech,” inProc. Int. Conf. Learn. Represent. (ICLR), 2021

    2021

  17. [17]

    HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 17 022–17 033

    2020

  18. [18]

    Improved techniques for training GANs,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Rad- ford, and X. Chen, “Improved techniques for training GANs,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 29, 2016, pp. 2234–2242

    2016

  19. [19]

    Match- ing novel face and voice identity using static and dynamic facial images,

    H. M. J. Smith, A. K. Dunn, T. Baguley, and P. C. Stacey, “Match- ing novel face and voice identity using static and dynamic facial images,”Attention, Perception, & Psychophysics, vol. 78, no. 3, pp. 868–879, 2016

    2016

  20. [20]

    Matching voice and face iden- tity from static images,

    L. W. Mavica and E. Barenholtz, “Matching voice and face iden- tity from static images,”Journal of Experimental Psychology: Human Perception and Performance, vol. 39, no. 2, pp. 307–312, 2013

    2013

  21. [21]

    Face2Speech: Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image,

    S. Goto, K. Onishi, Y . Saito, K. Tachibana, and K. Mori, “Face2Speech: Towards multi-speaker text-to-speech synthesis using an embedding vector predicted from a face image,” inProc. Interspeech 2020, 2020, pp. 1321–1325

    2020

  22. [22]

    LRS3-TED: A large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: A large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

    Pith/arXiv arXiv 2018

  23. [23]

    Imaginary voice: Face- styled diffusion model for text-to-speech,

    J. Lee, J. S. Chung, and S.-W. Chung, “Imaginary voice: Face- styled diffusion model for text-to-speech,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2023, pp. 1–5

    2023

  24. [24]

    Face-StyleSpeech: Enhancing zero-shot speech synthesis from face images with improved face- to-speech mapping,

    M. Kang, W. Han, and E. Yang, “Face-StyleSpeech: Enhancing zero-shot speech synthesis from face images with improved face- to-speech mapping,”arXiv preprint arXiv:2311.05844, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Emotional face-to-speech,

    J. Ye, B. Cao, and H. Shan, “Emotional face-to-speech,”arXiv preprint arXiv:2502.01046, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    V oxCeleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep speaker recognition,” inProc. Interspeech 2018, 2018, pp. 1086– 1090

    2018

  27. [27]

    Representation learning with contrastive predictive coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    Pith/arXiv arXiv 2018

  28. [28]

    Deep canonical correlation analysis,

    G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” inProc. Int. Conf. Mach. Learn. (ICML), 2013, pp. 1247–1255

    2013

  29. [29]

    Dimensionality reduction by learning an invariant mapping,

    R. Hadsell, S. Chopra, and Y . LeCun, “Dimensionality reduction by learning an invariant mapping,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2, 2006, pp. 1735–1742

    2006

  30. [30]

    Harnessing frozen unimodal en- coders for flexible multimodal alignment,

    M. Maniparambil, R. Akshulakov, Y . A. D. Djilali, S. Narayan, A. Singh, and N. E. O’Connor, “Harnessing frozen unimodal en- coders for flexible multimodal alignment,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025, pp. 29 847– 29 857

    2025

  31. [31]

    Both MOS pre- dictors are English-biased, so Spanish values are only indica- tive

    (signal quality), plus SECSaudio (identity). Both MOS pre- dictors are English-biased, so Spanish values are only indica- tive. Table 4 reports means over synthesized utterances of un- seen identities (inference atα=0.5); the reported checkpoint epoch was selected on this unseen development split, a mild optimism we declare. Ground-truth recordings are an...

  32. [32]

    Wav2CLIP: Learning robust audio representations from CLIP,

    H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2CLIP: Learning robust audio representations from CLIP,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2022, pp. 4563–4567

    2022

  33. [33]

    Relational knowledge dis- tillation,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge dis- tillation,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recog- nit. (CVPR), 2019, pp. 3967–3976

    2019

  34. [34]

    LibriTTS: A corpus derived from LibriSpeech for text-to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” inProc. Interspeech 2019, 2019, pp. 1526–1530

    2019

  35. [35]

    FaceNet: A unified embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 815–823

    2015

  36. [36]

    VG- GFace2: A dataset for recognising faces across pose and age,

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “VG- GFace2: A dataset for recognising faces across pose and age,” in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit. (FG), 2018, pp. 67–74

    2018

  37. [37]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” inProc. Int. Conf. Learn. Represent. (ICLR), 2019

    2019

  38. [38]

    Generalized end-to-end loss for speaker verification,

    L. Wan, Q. Wang, A. Papir, and I. Lopez Moreno, “Generalized end-to-end loss for speaker verification,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2018, pp. 4879–4883

    2018

  39. [39]

    UTMOS: UTokyo-SaruLab system for V oice- MOS challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab system for V oice- MOS challenge 2022,” inProc. Interspeech 2022, 2022, pp. 4521–4525

    2022

  40. [40]

    NISQA: A deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,

    G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “NISQA: A deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,” inProc. Inter- speech 2021, 2021, pp. 2127–2131

    2021