Pith. sign in

REVIEW 4 major objections 4 minor 34 references

MuteSwap: Visual-informed Silent Video Identity Conversion

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that silent lip videos plus a few facial images suffice to generate intelligible speech in an unseen target speaker's voice, with identity conversion matching or beating audio-input methods and staying stable under noise.

desk verdict MuteSwap defines a genuinely new task and demonstrates noise robustness, but its identity metrics are compromised by training-evaluation encoder overlap and the clean-condition claims are narrower than stated. read the letter →

arxiv 2507.00498 v3 pith:E25LKWZK submitted 2025-07-01 cs.SD cs.CVcs.LGcs.MMeess.AS

Yifan Liu , Yu Fang , Zhouhan Lin This is my paper
classification cs.SDcs.CVcs.LGcs.MMeess.AS
keywords silentface-basedvoiceconversionzero-shotidentityface-voicealignmentcontrastivelearningmutualinformationminimizationvideo-to-speechsynthesisnoise-robustspeech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Silent Face-based Voice Conversion (SFVC) is feasible: given only the lip-region frames of a silent video and a few images of a target speaker, a model can produce intelligible speech that carries the video's linguistic content in the target speaker's voice. The authors claim MuteSwap is the first system to do this zero-shot, meaning target speakers need not appear in training. If true, voice conversion no longer requires clean audio from the source, which makes it robust in noisy environments where audio-based face-voice conversion fails. The paper also claims that in clean conditions MuteSwap's identity conversion matches or beats audio-driven baselines.

What carries the argument

The identity transfer rests on two training mechanisms added to a video-to-speech backbone: the AF-CLIP loss, a symmetric contrastive loss (Eq. 1-3) that pulls facial and audio identity embeddings of the same speaker together in a shared representation space, and vCLUB-S mutual-information upper-bound minimization, implemented with a variational Gaussian estimator $q_\theta$ trained by alternating E/M steps, which pushes the facial identity embedding $\mathrm{EI}_f$ to be statistically independent of the content embedding $\mathrm{E}_C$. These losses run jointly with the L1 Mel-spectrogram reconstruction loss $L_{\mathrm{Rec}}$ in a single training stage; at inference the speech encoder and estimator are discarded and identity is swapped simply by feeding different facial images.

What would settle it

A concrete check: build a hold-out set of speakers whose faces are near-identical (e.g., twins) but whose voices are measurably distinct, and measure whether MuteSwap's converted speech is classified as the correct target twin by a speaker-verification encoder. If the encoder fails to separate the twins while audio-based FVC succeeds, the face-only identity signal is not sufficient for true zero-shot identity conversion in that regime.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a frozen facial-identity encoder can be aligned with a speech-identity encoder through a bidirectional contrastive loss (AF-CLIP), and that the resulting visual identity embedding, when disentangled from lip content by minimizing a mutual-information upper bound, transfers speaker identity into synthesized speech without any audio. The evidence is that MuteSwap reaches 4.63% EER on LRS3 identity-conversion pairs, lower than the 22.37% of FVMVC, while holding WER at 44.13% under babble and siren noise, where FVMVC's WER degrades to 81.55% and SP-FaceVC's to 95.47%. This is the claim that visual cues alone suffice for both content recovery and zero-shot identity transfer.

Load-bearing premise

The load-bearing premise is that a person's face, photographed from a few frames, carries enough information about how that person's voice sounds that a model trained on face-voice pairs can transfer identity to a speaker it has never heard.

Editorial extensions

If this is right

  • A silent video of one person's lips plus a few photos of another person yield speech in the second person's voice, with no audio recording of either needed.
  • The approach keeps intelligibility when the source audio track is corrupted by noise, because the source audio is never used.
  • Vanilla video-to-speech synthesis also improves when facial images guide the model, so the method doubles as a general V2S system.
  • Identity can be continuously interpolated between two speakers by mixing facial embeddings, enabling gradual voice blending rather than binary switching.
  • Because the model generalizes to VoxCeleb2 without fine-tuning, the conversion recipe transfers out of the training domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The strongest interpretation is that face-voice correlation is being exploited as a learnable prior; if that correlation is weak for certain speaker populations (e.g., identical twins, voice-changing conditions), zero-shot identity transfer will degrade, so real deployments should test per-cohort.
  • The identity-related gains could partly reflect the evaluation speaker encoder's own face-voice association biases; a cleaner test would use a hold-out speaker set with paired face-voice verification.
  • The interpolation experiment suggests identity is being treated as a continuous control variable; one testable extension is semantic voice attribute control (pitch register, breathiness) by steering the facial embedding along interpretable directions.
  • If the method is deployed, it turns a single photo plus a silent video into a speaker impersonation tool; that should surface in release decisions alongside the paper's technical benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Silent Face-based Voice Conversion (SFVC), a task in which speech is synthesized from a silent lip-region video of a source speaker while the speaker identity is transferred from static facial images of a target speaker, with no acoustic input at inference. The proposed framework, MuteSwap, combines a DiVISe-style video-to-speech pipeline with two additional training objectives: an AF-CLIP contrastive loss that aligns facial and audio identity embeddings, and a vCLUB-S mutual-information upper-bound loss intended to disentangle visual identity from lip content. Experiments on LRS3 and VoxCeleb2 report that MuteSwap remains intelligible under source-audio noise where audio-based FVC methods collapse, achieves identity-conversion metrics comparable to or better than FVC baselines in clean conditions, and improves vanilla V2S synthesis over DiVISe. The paper claims to be the first zero-shot identity conversion methodology for SFVC.

Significance. If the central claims are substantiated, the paper would introduce a useful new task formulation and demonstrate that face images alone can carry enough speaker-identity information for zero-shot voice conversion, with particular practical value in degraded-audio settings. The strongest evidence is Table 1, where FVMVC's WER degrades from 27.00% to 81.55% under source noise while MuteSwap stays at 44.13%, and the DET-curve analysis in the appendix is a nice visualization. However, the identity-conversion evaluation is weakened by a train/evaluation circularity: the Resemblyzer/Wan et al. (2018) encoder used to compute PSH, PSD, and EER is the same frozen encoder family that the AF-CLIP loss explicitly aligns the facial embeddings to. The subjective MOS-FVM result is too small and its confidence intervals overlap with FVMVC's. The manuscript also omits ID-FaceVC, the most closely related recent FVC baseline, and the provided code link points to DiVISe rather than to a MuteSwap release, which prevents direct replication. The noisy-robustness contribution is solid, but the clean-condition identity conversion claims need stronger, more independent evaluation.

major comments (4)
  1. [§6.1, §4.1, Eqs. (1)–(3)] The identity-conversion metrics are partially circular with the training objective. Section 6.1 states that the speech encoder is 'directly adopted from Resemblyzer (Wan et al. 2018)' and frozen, and Section 4.1 defines PSH, PSD, and EER using a speaker encoder from Wan et al. (2018). The AF-CLIP loss in Eqs. (1)–(3) trains the facial identity embedding to match exactly this class of audio speaker embeddings via contrastive learning. Thus the large EER gap in Table 2 (4.63% for MuteSwap vs. 22.37% for FVMVC on LRS3) could reflect the model's alignment to the evaluation function rather than a genuinely separable voice identity. To support the claim that visual cues alone suffice for identity transfer, the authors should re-evaluate Table 2 and the ablations in Table 5 with an independent speaker encoder (e.g., ECAPA-TDNN, WavLM, or a different pretrained verification model) and, if possible, with a larger perceptual face-voice matching test. Without this, the central identity-conversion claim is not fully established.
  2. [Table 2, §4.6] The ablation results are hard to interpret because the 'Baseline' model, trained only with the V2S reconstruction loss and without AF-CLIP or the MI estimator, already achieves an EER of 6.00% on LRS3, far below every FVC method (FVMVC 22.37%). This suggests that much of the identity-transfer ability comes from the pretrained facial encoder or from the way face images are supplied at inference, rather than from the proposed AF-CLIP and MI objectives, which together reduce EER only from 6.00% to 4.63%. The paper should analyze this more carefully; for example, by ablating the facial encoder's pretraining or by reporting identity metrics for a version that uses a randomly initialized facial encoder. As written, the attribution of the identity-conversion success to the proposed losses is not commensurate with the small ablation differences.
  3. [§2.1, Tables 1–2] ID-FaceVC (Rong and Liu 2024), described in the related work as a recent FVC method that explicitly minimizes mutual information between speaker identity and content, is missing from all experimental comparisons. Since ID-FaceVC is the most relevant baseline for both the task and the technical approach, its omission makes it impossible to assess whether MuteSwap's disentanglement strategy provides any advantage over an audio-based FVC method that already uses mutual-information minimization. The authors should add ID-FaceVC to Tables 1 and 2 (or explain why it cannot be included) and discuss the comparison in Section 4.
  4. [Abstract, §1, §4.3] The statement that MuteSwap achieves 'comparable or better' results in clean settings should be scoped to identity-conversion metrics. In Table 1, under clean source conditions, FVMVC achieves WER 27.00% and NISQA 3.620, whereas MuteSwap reports WER 44.13% and NISQA 3.220; MuteSwap is substantially worse on content accuracy and speech quality. Section 1 later phrases the claim as 'identity conversion performance comparable to or better,' which is accurate, but the abstract's phrase 'impressive performance in both speech synthesis and identity conversion' risks overstating the clean-condition speech-synthesis result. Please make the scope explicit throughout.
minor comments (4)
  1. [Code link (Abstract/§6)] The Code URL in the abstract points to https://github.com/PussyCat0700/DiVISe, which is the repository of the baseline method, not of MuteSwap. If a MuteSwap release is not yet available, this should be stated explicitly; as written, a reader cannot reproduce or inspect the proposed method.
  2. [§4.2, Tables 2–4] Subjective MOS-FVM and MOS-SQ are based on 15 volunteers. The confidence intervals for MOS-FVM (MuteSwap 3.09±0.20 vs. FVMVC 2.87±0.19) overlap, so the perceptual advantage is not statistically decisive; the paper should avoid presenting this as a strong confirmation.
  3. [§4.2, Table 1 vs. Table 3] The WER for MuteSwap is reported as 44.13% in Table 1 and 38.53% in Table 3 on LRS3. Please clarify whether these come from different evaluation subsets, different test conditions, or different versions of the model, since the current presentation makes the numbers appear inconsistent.
  4. [Throughout] There are several typographical issues, including 'V oice' in the abstract and Section 1, 'the the' in Section 3.5 ('simply swapping the the speaker images'), and inconsistent spacing around λ_MI and λ_CLIP in Eq. (8). A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

Identity-conversion metrics are partly circular: AF-CLIP trains the facial embedding to match the same frozen Wan/Resemblyzer speaker encoder later used for PSH/PSD/EER, while the noise-robustness and human MOS evidence remain independent.

  1. fitted input called prediction [Section 3.1 (AF-CLIP, Eqs. 1-3); Section 4.1 (Identity Conversion, PSD/PSH/EER); Section 6.1 (Modules)]
    "The speech encoder is directly adopted from Resemblyzer (Wan et al. 2018). All three encoders are frozen during training. ... We compute the cosine similarity between the ground truth and synthesized audio using a speaker encoder (Wan et al. 2018). ... We average the cosine similarity among these pairs, which is termed the Paired Speaker Diversity (PSD) metric. ... Following Liu, Fang, and Lin (2025); Shi, Mohamed, and Hsu (2022), we compute the Equal Error Rate (EER) using the cosine similarities of the positive and negative pairs."

    The AF-CLIP loss in Eqs. (1)-(3) explicitly optimizes cosine similarity between facial identity embeddings EIf and audio identity embeddings EIa, where EIa is produced by the frozen Wan/Resemblyzer speech encoder described in Section 6.1. The identity-conversion metrics PSH, PSD, and EER are then computed as cosine similarities in speaker-embedding space, with the only speaker encoder specified in Section 4.1 being the same Wan et al. (2018) encoder. Consequently, Table 2's EER advantage for MuteSwap (4.63%) over FVMVC (22.37%) and the ablation result that removing AF-CLIP raises EER from 4.81% to 6.00% are partly attributable to the model being trained to place facial embeddings at the target speaker's Wan-embedding location, then being evaluated with that same encoder.

full rationale

The central identity-conversion claim is partially circular because the AF-CLIP training objective (Eqs. 1-3) aligns the visual identity embedding EIf with the audio identity embedding EIa derived from the frozen Resemblyzer/Wan et al. (2018) speech encoder, and the PSH/PSD/EER metrics are computed with the same Wan-based speaker encoder. The reported improvements in identity metrics therefore partly measure how well the model matches the exact embedding function it was trained against, not purely whether visual cues transfer identity to unseen speakers. However, the circularity is not total: the synthesized waveform is not directly the optimized embedding, and the paper's noise-robustness results (NISQA and WER in Table 1) and the subjective MOS-FVM test are independent of this encoder alignment. The self-citations to the authors' prior DiVISe work for the V2S pipeline and vocoder fine-tuning are used as building blocks rather than as a uniqueness or forbidden-alternative argument, so they are not load-bearing circularity. On balance, the paper has genuine independent content, but its headline identity-conversion numbers are inflated by the shared Wan encoder between training and evaluation, warranting a moderate circularity score of 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it is an empirical ML system built from established components. The main unverified inputs are the domain assumptions that faces carry voice identity and lip motion carries linguistic content, plus the choice of frozen encoder features.

free parameters (3)
  • lambda_CLIP = 0.1
    Weight on the AF-CLIP loss in Eq. 8, chosen by hand to balance identity alignment against reconstruction.
  • lambda_MI = 0.01
    Weight on the mutual information loss in Eq. 8, taken from Wang et al. (2021) because larger values caused training instability.
  • number of sampled facial images = 16
    Chosen from the appendix sweep in Figure 6; performance saturates around this number.
assumptions (4)
  • domain assumption Facial appearance correlates with vocal identity strongly enough for zero-shot identity transfer.
    Invoked in Section 1 when the paper claims facial features correlate closely with vocal characteristics, and used to justify using face images as the sole identity source.
  • domain assumption Lip-region video conveys sufficient linguistic content for intelligible speech reconstruction.
    Inherited from the V2S literature in Section 2.2 and used as the basis for the reconstruction loss in Eq. 7, with no acoustic content supervision at inference.
  • standard math The vCLUB upper bound with a Gaussian variational estimator is a valid and sufficiently tight mutual information bound for the learned embeddings.
    The inequality comes from Cheng et al. (2020), but using it as an optimization target assumes the estimator in Eq. 6 is close enough to the true conditional distribution for the bound to drive meaningful disentanglement.
  • domain assumption Frozen pretrained encoders, specifically FaRL, AV-HuBERT, and Resemblyzer, provide features that support cross-modal identity alignment and content reconstruction.
    Stated in Section 6.1: all three encoders are frozen, and the trainable adapters must operate in their fixed feature spaces.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MuteSwap: Visual-informed Silent Video Identity Conversion." pith.science (2026). https://pith.science/paper/E25LKWZK

@misc{pith2026250700498,
  author       = {Pith},
  title        = {Pith review of: MuteSwap: Visual-informed Silent Video Identity Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E25LKWZK}},
  note         = {Machine review of arXiv:2507.00498}
}
read the original abstract

Conventional voice conversion modifies voice characteristics from a source speaker to a target speaker, relying on audio input from both sides. However, this process becomes infeasible when clean audio is unavailable, such as in silent videos or noisy environments. In this work, we focus on the task of Silent Face-based Voice Conversion (SFVC), which does voice conversion entirely from visual inputs. i.e., given images of a target speaker and a silent video of a source speaker containing lip motion, SFVC generates speech aligning the identity of the target speaker while preserving the speech content in the source silent video. As this task requires generating intelligible speech and converting identity using only visual cues, it is particularly challenging. To address this, we introduce MuteSwap, a novel framework that employs contrastive learning to align cross-modality identities and minimize mutual information to separate shared visual features. Experimental results show that MuteSwap achieves impressive performance in both speech synthesis and identity conversion, especially under noisy conditions where methods dependent on audio input fail to produce intelligible results, demonstrating both the effectiveness of our training approach and the feasibility of SFVC.

Figures

Figures reproduced from arXiv: 2507.00498 by the authors.

Figure 1
Figure 1. FVC (left) and the proposed SFVC task (right). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of log Mel-spectrograms converted from video sources with noisy sound track using FVC and SFVC. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. MuteSwap pipeline. For training, speaker identity is represented by facial images cropped from silent videos. At [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: AF-CLIP loss enforcing symmetrical alignment between the two modal￾ities. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Identity Embedding interpolation. The mixing ratio for target speaker facial embedding increases from left to right. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: DET Curve on LRS3 test pairs. outperforming the Baseline (6.00%) and significantly sur￾passing prior FVC methods such as SP-FaceVC (Weng, Shuai, and Cheng 2023) (23.81%) and FVMVC (Sheng et al. 2023) (42.00%). This demonstrates the effectiveness of MuteSwap in speaker …
Figure 6
Figure 6. Figure 6: The effect of the number of maximum frames used [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 8
Figure 8. Figure 8: Identity Embedding interpolation results. The mixing ratio for target speaker facial embedding increases from left to [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 22 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    S.; and Zisserman, A

    Afouras, T.; Chung, J. S.; and Zisserman, A. 2018. LRS3-TED: a large-scale dataset for visual speech recognition. ArXiv, abs/1809.00496

    arXiv 2018

  4. [4]

    Cheng, P.; Hao, W.; Dai, S.; Liu, J.; Gan, Z.; and Carin, L. 2020. CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information. arXiv:2006.12013

    work page Pith review arXiv 2020

  5. [5]

    Choi, J.; Hong, J.; and Ro, Y. M. 2023. DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding. arXiv:2308.07787

    work page Pith review arXiv 2023

  6. [6]

    Choi, J.; Kim, M.; and Ro, Y. M. 2023. Intelligible Lip-to-Speech Synthesis with Speech Units. arXiv:2305.19603

    arXiv 2023

  7. [7]

    S.; Nagrani, A.; and Zisserman, A

    Chung, J. S.; Nagrani, A.; and Zisserman, A. 2018. VoxCeleb2: Deep Speaker Recognition. In Interspeech

    work page 2018

  8. [8]

    Gulati, A.; Qin, J.; Chiu, C.-C.; Parmar, N.; Zhang, Y.; Yu, J.; Han, W.; Wang, S.; Zhang, Z.; Wu, Y.; and Pang, R. 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. ArXiv, abs/2005.08100

    arXiv 2020

Show all 34 references
  1. [9]

    A.; Nawaz, S.; Liaqat, M

    Hannan, A.; Manzoor, M. A.; Nawaz, S.; Liaqat, M. I.; Schedl, M.; and Noman, M. 2025. PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association. arXiv:2505.17002

    2025 arXiv

  2. [10]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770--778

    2016

  3. [11]

    Hsu, W.-N.; Remez, T.; Shi, B.; Donley, J.; and Adi, Y. 2023. ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Regeneration. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18796--18806

    2023

  4. [12]

    Kim, M.; Hong, J.; and Ro, Y. M. 2023. Lip-to-Speech Synthesis in the Wild with Multi-Task Learning. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5

    2023

  5. [13]

    Kong, J.; Kim, J.; and Bae, J. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXiv:2010.05646

    2020 arXiv

  6. [14]

    Lee, J.; Oh, Y.; Hwang, I.; and Lee, K. 2024. Hear Your Face: Face-based voice conversion with F0 estimation. In Interspeech 2024, 4378--4382

    2024

  7. [15]

    Li, J.; Tu, W.; and Xiao, L. 2023. Freevc: Towards High-Quality Text-Free One-Shot Voice Conversion. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5

    2023

  8. [16]

    Liu, Y.; Fang, Y.; and Lin, Z. 2025. DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility. arXiv:2503.05223

    2025 arXiv

  9. [17]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  10. [18]

    Lu, H.-H.; Weng, S.-E.; Yen, Y.-F.; Shuai, H.-H.; and Cheng, W.-H. 2021. Face-based Voice Conversion: Learning the Voice behind a Face. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, 496–505. New York, NY, USA: Association for Computing Machiner...

    2021

  11. [19]

    W.; and Pantic, M

    Mira, R.; Haliassos, A.; Petridis, S.; Schuller, B. W.; and Pantic, M. 2022. SVTS: Scalable Video-to-Speech Synthesis. arXiv:2205.02058

    2022 arXiv

  12. [20]

    Mittag, G.; Naderi, B.; Chehadi, A.; and M \"o ller, S. 2021. NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets. In Interspeech

    2021

  13. [21]

    R.; Mukhopadhyay, R.; Namboodiri, V.; and Jawahar, C

    Prajwal, K. R.; Mukhopadhyay, R.; Namboodiri, V.; and Jawahar, C. V. 2020 a . Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis. arXiv:2005.08209

    2020 arXiv

  14. [22]

    R.; Mukhopadhyay, R.; Namboodiri, V

    Prajwal, K. R.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020 b . A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, 484–492. New York, NY, USA: Association for ...

    2020

  15. [23]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  16. [24]

    Rong, Y.; and Liu, L. 2024. Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion. arXiv:2409.00700

    2024 arXiv

  17. [25]

    S.; Khan, M

    Saeed, M. S.; Khan, M. H.; Nawaz, S.; Yousaf, M. H.; and Bue, A. D. 2021. Fusion and Orthogonal Projection for Improved Face-Voice Association. arXiv:2112.10483

    2021 arXiv

  18. [26]

    Sheng, Z.-Y.; Ai, Y.; Chen, Y.-N.; and Ling, Z.-H. 2023. Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignment. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, 8443–8452. New York, NY, USA: Association for Computing Machin...

    2023

  19. [27]

    Shi, B.; Hsu, W.-N.; Lakhotia, K.; and rahman Mohamed, A. 2022. Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction. ArXiv, abs/2201.02184

    2022 arXiv

  20. [28]

    Shi, B.; Mohamed, A.; and Hsu, W.-N. 2022. Learning lip-based audio-visual speaker embeddings with av-hubert. arXiv preprint arXiv:2205.07180

    2022 arXiv

  21. [29]

    Snyder, D.; Chen, G.; and Povey, D. 2015. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484

    2015 arXiv

  22. [30]

    Wan, L.; Wang, Q.; Papir, A.; and Moreno, I. L. 2018. Generalized End-to-End Loss for Speaker Verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4879--4883

    2018

  23. [31]

    T.; Chen, X.; Liu, X.; and Meng, H

    Wang, D.; Deng, L.; Yeung, Y. T.; Chen, X.; Liu, X.; and Meng, H. 2021. VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-Shot Voice Conversion . In Proc. Interspeech 2021, 1344--1348

    2021

  24. [32]

    Weng, S.-E.; Shuai, H.-H.; and Cheng, W.-H. 2023. Zero-Shot Face-Based Voice Conversion: Bottleneck-Free Speech Disentanglement in the Real-World Scenario. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11): 13718--13726

    2023

  25. [33]

    Yemini, Y.; Shamsian, A.; Bracha, L.; Gannot, S.; and Fetaya, E. 2024. LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading. arXiv:2306.03258

    2024 arXiv

  26. [34]

    Zheng, Y.; Yang, H.; Zhang, T.; Bao, J.; Chen, D.; Huang, Y.; Yuan, L.; Chen, D.; Zeng, M.; and Wen, F. 2022. General Facial Representation Learning in a Visual-Linguistic Manner. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18676--18688

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.