{"id":"916cc10b-0490-4292-a63e-e38264ee978c","arxiv_id":"2507.00498","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A single-stage model performs zero-shot voice conversion from silent lip video and target face images, with no acoustic input at inference.","lead":"MuteSwap generates speech from a silent video of one person's lips while making the voice sound like a different person shown only in photos. It introduces silent face-based voice conversion and stays intelligible when the source audio track is noisy or missing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Identity-conversion metrics may be circular: the evaluator speaker encoder (Resemblyzer/Wan et al. 2018) is the same frozen encoder used by the AF-CLIP training loss, so Table 2's large gains could reflect train-eval alignment rather than genuine voice identity transfer.","rationale":"The reader's verdict is CONDITIONAL, and the reader already highlighted that the objective identity metrics share the frozen speaker encoder used in training. My independent reading converges on the same point as the single most load-bearing issue. The paper's headline claim is that visual cues alone can drive identity conversion, and the strongest quantitative support is Table 2, whose EER is dramatically lower for MuteSwap than for FVC methods. That support depends on a general speaker-embedding model, Resemblyzer, which is also the exact embedding space into which AF-CLIP maps facial identity. Since all three encoders (facial, content, speech) are frozen and only adapters are trained, the learned adapters can essentially learn to project facial images into the Resemblyzer audio space; at inference, converted speech is then scored by the same space, making high identity-similarity scores somewhat self-fulfilling. This is not an accusation of dishonesty: the authors disclose the modules and the frozen speech encoder in the appendix. It is a genuine methodological gap. The noise-robustness result (Table 1) and the human MOS-FVM and MOS-SQ results are less affected by this circularity and provide real support for the core pipeline; the interpolation experiment also suggests identity control is not vacuous. However, the comparative claim 'comparable to or better than FVC methods in clean settings' is not established without an independent identity evaluator. Because the reader already conditioned acceptance on replication and cleaner evaluation, my concern does not change the verdict; it sharpens the specific condition. I did not find a different, more severe logical flaw in the architecture or training objective: the MI-minimization and CLIP losses are applied to the adapters, the single-stage training is plausible, and the noise experiment is well-designed. The code availability and missing baseline are additional weaknesses but secondary to the metric circularity.","tokens_in":14500,"tokens_out":3112,"duration_ms":42726,"concrete_test":"Recompute Table 2's PSH/PSD/EER on the same converted samples using an independent speaker embedding model not used anywhere in training—e.g., SpeechBrain ECAPA-TDNN trained on VoxCeleb, or WavLM-TDNN—for MuteSwap, FVMVC, and the SFVC baseline on both LRS3 and VoxCeleb2. Also report SECS with the same independent encoder. If MuteSwap no longer beats FVMVC/baseline, or the EER gap narrows substantially, the objective identity-conversion claim is an artifact of evaluator alignment. A larger MOS-FVM test with >50 listeners could corroborate, but the independent-encoder check is the decisive, low-cost test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MuteSwap transfers speaker identity from facial images is most at risk from a train/evaluation circularity. Section 4.1 defines the identity metrics PSH/PSD/EER using a speaker encoder (Wan et al. 2018), and Section 6.1 states that the speech encoder used in training is 'directly adopted from Resemblyzer (Wan et al. 2018)' and remains frozen. The AF-CLIP loss (Eqs. 1–3) explicitly aligns the facial identity embedding EIf to these Resemblyzer audio embeddings through contrastive learning. Consequently, the evaluation encoder is the very function the model was trained to match. If an independent speaker encoder were used, the EER gap in Table 2 (4.63% for MuteSwap vs. 22.37% for FVMVC on LRS3) could shrink or disappear. The 15-subject MOS-FVM (3.09 vs. 2.87) offers some independent human evidence, but its small size does not resolve the circularity. The claim that visual cues alone suffice for identity transfer therefore rests on metrics partially optimized by construction. The code repository points to DiVISe rather than a MuteSwap release, preventing direct replication, and ID-FaceVC is missing as the most relevant baseline, but the circular-metric issue is the load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Silent Face-based Voice Conversion (SFVC), a task in which speech is synthesized from a silent lip-region video of a source speaker while the speaker identity is transferred from static facial images of a target speaker, with no acoustic input at inference. The proposed framework, MuteSwap, combines a DiVISe-style video-to-speech pipeline with two additional training objectives: an AF-CLIP contrastive loss that aligns facial and audio identity embeddings, and a vCLUB-S mutual-information upper-bound loss intended to disentangle visual identity from lip content. Experiments on LRS3 and VoxCeleb2 report that MuteSwap remains intelligible under source-audio noise where audio-based FVC methods collapse, achieves identity-conversion metrics comparable to or better than FVC baselines in clean conditions, and improves vanilla V2S synthesis over DiVISe. The paper claims to be the first zero-shot identity conversion methodology for SFVC.","tokens_in":14785,"tokens_out":4516,"duration_ms":48773,"significance":"If the central claims are substantiated, the paper would introduce a useful new task formulation and demonstrate that face images alone can carry enough speaker-identity information for zero-shot voice conversion, with particular practical value in degraded-audio settings. The strongest evidence is Table 1, where FVMVC's WER degrades from 27.00% to 81.55% under source noise while MuteSwap stays at 44.13%, and the DET-curve analysis in the appendix is a nice visualization. However, the identity-conversion evaluation is weakened by a train/evaluation circularity: the Resemblyzer/Wan et al. (2018) encoder used to compute PSH, PSD, and EER is the same frozen encoder family that the AF-CLIP loss explicitly aligns the facial embeddings to. The subjective MOS-FVM result is too small and its confidence intervals overlap with FVMVC's. The manuscript also omits ID-FaceVC, the most closely related recent FVC baseline, and the provided code link points to DiVISe rather than to a MuteSwap release, which prevents direct replication. The noisy-robustness contribution is solid, but the clean-condition identity conversion claims need stronger, more independent evaluation.","major_comments":[{"comment":"The identity-conversion metrics are partially circular with the training objective. Section 6.1 states that the speech encoder is 'directly adopted from Resemblyzer (Wan et al. 2018)' and frozen, and Section 4.1 defines PSH, PSD, and EER using a speaker encoder from Wan et al. (2018). The AF-CLIP loss in Eqs. (1)–(3) trains the facial identity embedding to match exactly this class of audio speaker embeddings via contrastive learning. Thus the large EER gap in Table 2 (4.63% for MuteSwap vs. 22.37% for FVMVC on LRS3) could reflect the model's alignment to the evaluation function rather than a genuinely separable voice identity. To support the claim that visual cues alone suffice for identity transfer, the authors should re-evaluate Table 2 and the ablations in Table 5 with an independent speaker encoder (e.g., ECAPA-TDNN, WavLM, or a different pretrained verification model) and, if possible, with a larger perceptual face-voice matching test. Without this, the central identity-conversion claim is not fully established.","section":"§6.1, §4.1, Eqs. (1)–(3)"},{"comment":"The ablation results are hard to interpret because the 'Baseline' model, trained only with the V2S reconstruction loss and without AF-CLIP or the MI estimator, already achieves an EER of 6.00% on LRS3, far below every FVC method (FVMVC 22.37%). This suggests that much of the identity-transfer ability comes from the pretrained facial encoder or from the way face images are supplied at inference, rather than from the proposed AF-CLIP and MI objectives, which together reduce EER only from 6.00% to 4.63%. The paper should analyze this more carefully; for example, by ablating the facial encoder's pretraining or by reporting identity metrics for a version that uses a randomly initialized facial encoder. As written, the attribution of the identity-conversion success to the proposed losses is not commensurate with the small ablation differences.","section":"Table 2, §4.6"},{"comment":"ID-FaceVC (Rong and Liu 2024), described in the related work as a recent FVC method that explicitly minimizes mutual information between speaker identity and content, is missing from all experimental comparisons. Since ID-FaceVC is the most relevant baseline for both the task and the technical approach, its omission makes it impossible to assess whether MuteSwap's disentanglement strategy provides any advantage over an audio-based FVC method that already uses mutual-information minimization. The authors should add ID-FaceVC to Tables 1 and 2 (or explain why it cannot be included) and discuss the comparison in Section 4.","section":"§2.1, Tables 1–2"},{"comment":"The statement that MuteSwap achieves 'comparable or better' results in clean settings should be scoped to identity-conversion metrics. In Table 1, under clean source conditions, FVMVC achieves WER 27.00% and NISQA 3.620, whereas MuteSwap reports WER 44.13% and NISQA 3.220; MuteSwap is substantially worse on content accuracy and speech quality. Section 1 later phrases the claim as 'identity conversion performance comparable to or better,' which is accurate, but the abstract's phrase 'impressive performance in both speech synthesis and identity conversion' risks overstating the clean-condition speech-synthesis result. Please make the scope explicit throughout.","section":"Abstract, §1, §4.3"}],"minor_comments":[{"comment":"The Code URL in the abstract points to https://github.com/PussyCat0700/DiVISe, which is the repository of the baseline method, not of MuteSwap. If a MuteSwap release is not yet available, this should be stated explicitly; as written, a reader cannot reproduce or inspect the proposed method.","section":"Code link (Abstract/§6)"},{"comment":"Subjective MOS-FVM and MOS-SQ are based on 15 volunteers. The confidence intervals for MOS-FVM (MuteSwap 3.09±0.20 vs. FVMVC 2.87±0.19) overlap, so the perceptual advantage is not statistically decisive; the paper should avoid presenting this as a strong confirmation.","section":"§4.2, Tables 2–4"},{"comment":"The WER for MuteSwap is reported as 44.13% in Table 1 and 38.53% in Table 3 on LRS3. Please clarify whether these come from different evaluation subsets, different test conditions, or different versions of the model, since the current presentation makes the numbers appear inconsistent.","section":"§4.2, Table 1 vs. Table 3"},{"comment":"There are several typographical issues, including 'V oice' in the abstract and Section 1, 'the the' in Section 3.5 ('simply swapping the the speaker images'), and inconsistent spacing around λ_MI and λ_CLIP in Eq. (8). A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core noisy-robustness result is compelling and the task formulation is valuable. The main risk is that the identity-conversion metrics are entangled with the training loss through the shared Wan et al. (2018) speaker encoder; this needs to be addressed with an independent evaluation, not just a rebuttal. I also find it surprising that the two-line baseline achieves EER 6.00% without any identity-specific losses, which suggests the proposed losses may be less central than claimed. The missing ID-FaceVC baseline and the code-link issue should be straightforward to fix. If the authors can provide an independent-encoder evaluation and a proper baseline set, I would be willing to revisit the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading: it defines a real new task, Silent Face-based Voice Conversion (SFVC), where both content and identity come from video/face images with no audio; and its headline noise result is solid. But the identity conversion numbers are partly an artifact of using the same speaker encoder for training and evaluation, so treat the clean-condition comparisons with caution.\n\nThe new task is the paper's main contribution. Neither FVC nor V2S covers this combination, and the paper is upfront that the pieces are borrowed: CLIP-style contrastive learning, vCLUB mutual information bound, and their prior DiVISe V2S pipeline. That's fine. The method is a sensible single-stage integration, and zero-shot conversion is done by swapping facial images at inference.\n\nThe strongest evidence is Table 1. Under babble/siren noise, FVMVC's WER jumps from 27.00% to 81.55%, while MuteSwap stays at 44.13% because it never sees audio. That's a genuine and practically useful property.\n\nNow the soft spots. The identity metrics PSH/PSD/EER are computed with the Resemblyzer speaker encoder, which is the same frozen encoder the AF-CLIP loss trains the facial embedding to match. So the model is scored on the exact similarity function it was optimized against. That is not outright circular—the 15-subject MOS-FVM (3.09 vs 2.87) points the same direction—but it is a load-bearing evaluation flaw. A 15-person MOS cannot carry it on its own. Also, the V2S-only baseline already reaches 6.00% EER on LRS3, which is suspicious and suggests the test pairs are not very discriminative.\n\nOther issues: the most relevant FVC baseline, ID-FaceVC, is cited but never compared. The code link points to the prior DiVISe repo, not a MuteSwap release, so replication is impossible as submitted. And the 'comparable or better' clean claim is only about identity—MuteSwap's clean WER (44.13) is well behind FVMVC's clean WER (27.00). The paper should say that explicitly.\n\nWho is this for? Researchers in video-to-speech, voice conversion, and multimodal representation learning. It deserves a serious referee. I'd send it out, but the revision should demand an independent speaker encoder for identity evaluation, a comparison with ID-FaceVC, and actual MuteSwap code. Without those, the central identity claim remains an open question.","headline":"MuteSwap defines a genuinely new task and demonstrates noise robustness, but its identity metrics are compromised by training-evaluation encoder overlap and the clean-condition claims are narrower than stated.","tokens_in":15326,"tokens_out":5790,"would_cite":true,"duration_ms":58622,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that silent lip videos plus a few facial images suffice to generate intelligible speech in an unseen target speaker's voice, with identity conversion matching or beating audio-input methods and staying stable under noise.","keywords":["silent face-based voice conversion","zero-shot identity conversion","face-voice alignment","contrastive learning","mutual information minimization","video-to-speech synthesis","noise-robust speech synthesis"],"falsifier":"A concrete check: build a hold-out set of speakers whose faces are near-identical (e.g., twins) but whose voices are measurably distinct, and measure whether MuteSwap's converted speech is classified as the correct target twin by a speaker-verification encoder. If the encoder fails to separate the twins while audio-based FVC succeeds, the face-only identity signal is not sufficient for true zero-shot identity conversion in that regime.","tokens_in":14276,"feed_emoji":"🗣️","tokens_out":5116,"duration_ms":48238,"temperature":0.7,"pith_summary":"The paper tries to establish that Silent Face-based Voice Conversion (SFVC) is feasible: given only the lip-region frames of a silent video and a few images of a target speaker, a model can produce intelligible speech that carries the video's linguistic content in the target speaker's voice. The authors claim MuteSwap is the first system to do this zero-shot, meaning target speakers need not appear in training. If true, voice conversion no longer requires clean audio from the source, which makes it robust in noisy environments where audio-based face-voice conversion fails. The paper also claims that in clean conditions MuteSwap's identity conversion matches or beats audio-driven baselines.","feed_headline":"A face photo can voice a silent video as a new speaker","feed_subtitle":"MuteSwap recovers intelligible speech from lip motion and transfers identity zero-shot from facial images, even when audio is ruined.","key_machinery":"The identity transfer rests on two training mechanisms added to a video-to-speech backbone: the AF-CLIP loss, a symmetric contrastive loss (Eq. 1-3) that pulls facial and audio identity embeddings of the same speaker together in a shared representation space, and vCLUB-S mutual-information upper-bound minimization, implemented with a variational Gaussian estimator $q_\\theta$ trained by alternating E/M steps, which pushes the facial identity embedding $\\mathrm{EI}_f$ to be statistically independent of the content embedding $\\mathrm{E}_C$. These losses run jointly with the L1 Mel-spectrogram reconstruction loss $L_{\\mathrm{Rec}}$ in a single training stage; at inference the speech encoder and estimator are discarded and identity is swapped simply by feeding different facial images.","core_discovery":"On the paper's own terms, the central discovery is that a frozen facial-identity encoder can be aligned with a speech-identity encoder through a bidirectional contrastive loss (AF-CLIP), and that the resulting visual identity embedding, when disentangled from lip content by minimizing a mutual-information upper bound, transfers speaker identity into synthesized speech without any audio. The evidence is that MuteSwap reaches 4.63% EER on LRS3 identity-conversion pairs, lower than the 22.37% of FVMVC, while holding WER at 44.13% under babble and siren noise, where FVMVC's WER degrades to 81.55% and SP-FaceVC's to 95.47%. This is the claim that visual cues alone suffice for both content recovery and zero-shot identity transfer.","pith_inferences":["The strongest interpretation is that face-voice correlation is being exploited as a learnable prior; if that correlation is weak for certain speaker populations (e.g., identical twins, voice-changing conditions), zero-shot identity transfer will degrade, so real deployments should test per-cohort.","The identity-related gains could partly reflect the evaluation speaker encoder's own face-voice association biases; a cleaner test would use a hold-out speaker set with paired face-voice verification.","The interpolation experiment suggests identity is being treated as a continuous control variable; one testable extension is semantic voice attribute control (pitch register, breathiness) by steering the facial embedding along interpretable directions.","If the method is deployed, it turns a single photo plus a silent video into a speaker impersonation tool; that should surface in release decisions alongside the paper's technical benchmark."],"forward_implications":["A silent video of one person's lips plus a few photos of another person yield speech in the second person's voice, with no audio recording of either needed.","The approach keeps intelligibility when the source audio track is corrupted by noise, because the source audio is never used.","Vanilla video-to-speech synthesis also improves when facial images guide the model, so the method doubles as a general V2S system.","Identity can be continuously interpolated between two speakers by mixing facial embeddings, enabling gradual voice blending rather than binary switching.","Because the model generalizes to VoxCeleb2 without fine-tuning, the conversion recipe transfers out of the training domain."],"supporting_citations":[{"why":"Supplies the mutual-information-based disentanglement recipe (VQMIVC) and the variational Gaussian estimator that the MI loss is built on.","marker":"Wang et al. 2021"},{"why":"Supplies the CLIP contrastive alignment idea adapted as the AF-CLIP loss.","marker":"Radford et al. 2021"},{"why":"The DiVISe V2S pipeline, content encoder, blender, and vocoder fine-tuning that MuteSwap extends.","marker":"Liu, Fang, and Lin 2025"},{"why":"FVMVC, the strongest FVC baseline and source of the LRS3 top-200 subset and conversion test pairs.","marker":"Sheng et al. 2023"},{"why":"SP-FaceVC baseline used for noisy-condition comparison.","marker":"Weng, Shuai, and Cheng 2023"},{"why":"vCLUB-S, the contrastive log-ratio upper bound of mutual information used for identity-content disentanglement.","marker":"Cheng et al. 2020"},{"why":"AV-HuBERT pretrained visual and audio encoders used as frozen backbones.","marker":"Shi et al. 2022"},{"why":"Speaker encoder used for SECS and identity-conversion metrics (Resemblyzer).","marker":"Wan et al. 2018"},{"why":"LRS3 dataset used for training and evaluation.","marker":"Afouras, Chung, and Zisserman 2018"},{"why":"HiFi-GAN vocoder, fine-tuned on V2S outputs to produce the final waveform.","marker":"Kong et al. 2020"}],"fun_headline_variants":["Visual-only voice identity transfer from a still photo","Silent video speech with face-driven speaker identity","Face photo enables noise-robust speech conversion","Zero-shot identity conversion from facial images alone","MuteSwap: speech from lip motion and a face photo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a person's face, photographed from a few frames, carries enough information about how that person's voice sounds that a model trained on face-voice pairs can transfer identity to a speaker it has never heard.","fun_headline_variants_meta":{"raw":{"variants":["Visual-only voice identity transfer from a still photo","Silent video speech with face-driven speaker identity","Face photo enables noise-robust speech conversion","Zero-shot identity conversion from facial images alone","MuteSwap: speech from lip motion and a face photo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2645,"prompt_tokens":908,"completion_tokens":1737,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1664}},"tokens_in":524,"tokens_out":1737,"duration_ms":14091,"temperature":1.0,"reasoning_tokens":1664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:13:30.488392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: build a hold-out set of speakers whose faces are near-identical (e.g., twins) but whose voices are measurably distinct, and measure whether MuteSwap's converted speech is classified as the correct target twin by a speaker-verification encoder. If the encoder fails to separate the twins while audio-based FVC succeeds, the face-only identity signal is not sufficient for true zero-shot identity conversion in that regime.","supporting_citations":[{"cited_title":"T.; Chen, X.; Liu, X.; and Meng, H","cited_arxiv_id":null,"evidence_quote":"Supplies the mutual-information-based disentanglement recipe (VQMIVC) and the variational Gaussian estimator that the MI loss is built on."},{"cited_title":"DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility","cited_arxiv_id":"2503.05223","evidence_quote":"The DiVISe V2S pipeline, content encoder, blender, and vocoder fine-tuning that MuteSwap extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FVMVC, the strongest FVC baseline and source of the LRS3 top-200 subset and conversion test pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SP-FaceVC baseline used for noisy-condition comparison."},{"cited_title":"CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information","cited_arxiv_id":"2006.12013","evidence_quote":"vCLUB-S, the contrastive log-ratio upper bound of mutual information used for identity-content disentanglement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Speaker encoder used for SECS and identity-conversion metrics (Resemblyzer)."}],"review_version":1}