{"id":"4eef1f13-51ed-4798-af92-df649b4aa778","arxiv_id":"2505.22024","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A lip-to-speech model that predicts prosody from an audio prompt and content from lip-reading, then fuses them to synthesize speech, achieving strong benchmark results on LRS2 and LRS3.","lead":"RESOUND reconstructs speech from silent videos by separating the task into an acoustic branch for prosody and a semantic branch for content. The authors report state-of-the-art word error rates and naturalness on two standard benchmarks, though one of the key evaluation metrics is tied to the same recognizer used to supervise the model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"0.5s ground-truth audio prompt at test time leaks target content and prosody; Table 1 gains may be an artifact, not model quality.","rationale":"The reader's weakest assumption was ASR circularity: WER is computed with Auto-AVSR, the same model that supplies text supervision. This is a valid concern, but the paper includes a human MOS intelligibility test (Table 2) showing RESOUND at 3.86 vs. 2.98 for DiffV2S, which partly addresses the circularity worry. In contrast, the ground-truth audio prompt issue is more load-bearing because it affects all objective metrics and undermines the problem set-up itself. If the 0.5s prompt is taken from the same utterance, it provides the model with access to the target speech content, prosody, and voice identity. This could inflate WER, UTMOS, SECS, MAE_F0, and ESTOI simultaneously, making the SOTA claim in the strongest claim ('reduces WER to 20.06 from 27.69 and raises UTMOS to 3.002') depend on an unfair test-time information leak. The reader noted the prompt as a 'need' but did not elevate it to the primary weak assumption. We therefore disagree in emphasis, not in overall assessment: the verdict remains CONDITIONAL, pending a test that isolates the prompt effect. The proposed concrete test is straightforward and would settle whether the prompt is the true source of the gains.","tokens_in":8875,"tokens_out":4766,"duration_ms":53111,"concrete_test":"Re-evaluate RESOUND and all baselines on LRS3-TED under three conditions: (a) same-utterance 0.5s ground-truth prompt (as in the paper), (b) a 0.5s prompt from a different utterance of the same speaker, and (c) no audio prompt. If the WER/UTMOS/SECS advantages shrink or vanish in (b) or (c), the reported SOTA is an artifact of test-time ground-truth audio leakage. Report baselines re-run under the same conditions, not just published numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that a 0.5s segment is randomly extracted from the ground-truth audio to serve as the speaker prompt. At inference, for each test video, the model therefore receives a slice of the exact audio it is supposed to reconstruct. This leaks (i) speaker timbre, (ii) pitch/energy of the target utterance, and (iii) potentially partial lexical content if the slice overlaps words in the scored utterance. The Prosody Encoder and Timbre Encoder condition directly on this prompt, so all objective metrics (WER, UTMOS, SECS, MAE_F0, ESTOI) can be inflated relative to baselines that do not receive ground-truth audio. Table 1 appears to use published baseline numbers rather than re-running baselines under the same prompt condition; the paper only says 'Following [11]' for its own setup. If baselines lack this prompt, the SOTA conclusion is confounded. Even if baselines use prompts, a same-utterance prompt is a test-set information leak that violates the problem's premise of reconstructing missing audio from silent video. The WER circularity raised by the reader is real, but it is partially mitigated by the human MOS test (Table 2); the prompt leakage affects every metric and is not mitigated by any human evaluation, which also only used 20 samples per method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RESOUND, a lip-to-speech (L2S) system that decomposes speech reconstruction into an acoustic branch (prosody, timbre) and a semantic branch (linguistic content) motivated by source-filter theory. The acoustic branch predicts pitch and energy from a short audio prompt, while the semantic branch combines visual features from AV-HuBERT with text predictions from an L2T model and phoneme embeddings. The two streams are fused in a Spec-Ling Decoder that predicts both a mel-spectrogram and discrete speech units, followed by a vocoder. Experiments on LRS2-BBC and LRS3-TED report improved WER, UTMOS, SECS, and low-level metrics over prior methods, with ablations supporting the individual contributions. The manuscript includes a demo page and states that source code will be released.","tokens_in":9166,"tokens_out":3879,"duration_ms":44287,"significance":"If the results hold, RESOUND would be a valuable contribution: it is the first L2S system to explicitly apply source-filter decomposition, it combines multiple supervision signals (visual, phoneme, speech units, acoustic predictors) in a structured way, and it improves objective and subjective quality on large-scale datasets. The code release and the ablation studies are assets. However, the evaluation contains two load-bearing confounds: the WER metric uses the same ASR model that provides the L2T supervision, and the test-time speaker prompt is a slice of the ground-truth audio. These issues affect the validity of the central SOTA claim and must be addressed before the results can be fully trusted.","major_comments":[{"comment":"The intelligibility claim rests on WER computed by Auto-AVSR, which is the same model ([22]) used inside RESOUND as the L2T text predictor. Section 2.2.1 states that the L2T model generates textual predictions that are converted to phoneme embeddings, and these are used to supervise the semantic branch. Section 3.2 then evaluates WER with Auto-AVSR. This creates a circular evaluation: the model is trained to produce speech that this specific recognizer decodes well, and then measured with that same recognizer. The reported improvement from 27.69 to 20.06 on LRS3-TED may therefore reflect an overfit to Auto-AVSR rather than a general gain in intelligibility. The authors should report WER with an independent ASR not used in training (e.g., Whisper or an ESPnet model) and, ideally, human transcription evaluations on a larger set than the 20 samples in Table 2.","section":"§3.2, §2.2.1"},{"comment":"The test-time setup leaks information from the ground-truth audio. Section 3.3 says a 0.5s segment is randomly extracted from the ground-truth audio to serve as the audio prompt. At inference, the model therefore receives a slice of the exact utterance it must reconstruct. This leaks speaker timbre, pitch/energy, and possibly partial lexical content if the slice overlaps the scored portion. All objective metrics in Table 1 (WER, UTMOS, SECS, MAE_F0, ESTOI, MCD-DTW-SL) can be inflated relative to baselines that do not receive this prompt. The paper states only that its own setup follows [11]; it does not indicate whether the published baseline numbers were obtained under the same prompting condition. The SOTA conclusion is confounded unless all methods are re-run with identical prompts (or no prompt). The authors should either remove the ground-truth prompt, use a prompt from a different utterance/spaker, or re-run all baselines under the exact same condition and report the results with error bars.","section":"§3.3, Table 1"},{"comment":"Table 1 reports a single set of metric values without standard deviations, confidence intervals, or significance tests. Some differences are small (e.g., WER 20.06 vs 21.04 in LRS3-TED; SECS 0.777 vs 0.750 for RESOUND vs Intelligible L2S) and could easily arise from run-to-run variance. Without multiple seeds or bootstrap intervals, the claim that RESOUND 'consistently outperforms' baselines is not statistically supported. The paper should provide variance estimates and, where appropriate, pairwise significance tests.","section":"Table 1"},{"comment":"On LRS2-BBC, RESOUND has a substantially worse WER than LipVoicer (28.55 vs 17.04). The text acknowledges this and attributes it to noise in LRS2. But intelligibility is a core objective of L2S reconstruction; a method that loses on the primary intelligibility metric on one of the two benchmarks needs a more careful discussion of the trade-off. The paper should either demonstrate that the WER gap is offset by other metrics in a clear way, or temper the 'state-of-the-art' claim to specify the metric and dataset. As written, the claim in the abstract that the method 'consistently outperforms' is too strong.","section":"§4.1, Table 1 (LRS2-BBC)"}],"minor_comments":[{"comment":"The footnote in Section 4.1 says LSE-* metrics are omitted because some methods outperform ground truth. This is not a convincing reason on its own; LSE-D, LSE-C, and LSE-S are standard in L2S evaluation and are widely reported even when imperfect. Please either provide the scores and explain the discrepancy, or present an alternative synchronization metric, so readers can assess audio-visual alignment.","section":"§4.1, LSE-* footnote"},{"comment":"Table 2 reports MOS on 20 randomly synthesized samples per method. Please specify how the samples were selected (e.g., random utterance indices, seed), how many participants per condition, and whether the 95% confidence intervals are per-method. With only 20 samples and 30 participants, the results are informative but not very precise for a claimed SOTA difference.","section":"Table 2"},{"comment":"Equation (3) lists the loss weights, but there is no sensitivity analysis for these weights (λ_m=100, λ_p=0.1, λ_e=0.1, λ_u=0.01). Given that the paper argues for the importance of the acoustic branch and speech units, a brief experiment on the loss scales would strengthen the robustness of the method.","section":"§2.4"},{"comment":"The implementation details mention random cropping and flipping for the visual input, which are standard augmentations, but there is no mention of an inference-time augmentation or test-time averaging. Please specify the exact inference protocol (e.g., whether the random cropping is disabled at test time).","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The two major confounds (Auto-AVSR circularity and ground-truth audio prompt leakage) are central to the paper's contribution. They are fixable within the scope of a revision: re-running baselines under the same prompt condition, evaluating with an independent ASR, and re-benchmarking with proper variance reporting would address the issues. I do not see an internal inconsistency that would force rejection, but the evaluation must be substantially strengthened before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things about this paper. First, the architecture is genuinely new for L2S: splitting acoustic and semantic modeling in a source-filter style, with L2T+G2P for text, Conformer for fusion, and dual mel-plus-speech-unit decoding. The ablations show that each branch matters, and the human MOS test is a point in its favor despite the small sample. Second, the evaluation is not trustworthy as it stands. The WER is measured by Auto-AVSR, which is also the model providing text supervision in the semantic branch, and the 0.5s ground-truth audio prompt at test time leaks content and prosody of the exact utterance being reconstructed.\n\nThe prompt leakage is the bigger issue. Section 3.3 says they randomly extract a 0.5s segment from the ground-truth audio to serve as the speaker prompt. At inference, the Prosody and Timbre Encoders condition on that slice of the target audio. This can only inflate every reported metric, not just WER. The paper says 'Following [11]' for the setup, but Table 1 appears to compare against published baseline numbers; if those baselines were not evaluated with the same prompt condition, the gains are apples-to-oranges. Even if baselines had prompts, a same-utterance prompt violates the task premise of reconstructing missing audio from silent video.\n\nThe WER circularity compounds this: Auto-AVSR is both teacher and grader. The human MOS test partially mitigates it, but it used only 20 samples per method and does not address the prompt leak, which affects all objective metrics.\n\nThe paper also deserves credit for reporting LRS2 WER honestly, noting that LipVoicer beats it there. But the selective omission of LSE metrics is weak; saying some methods beat ground truth is not a good reason to drop them.\n\nFor peer review: I'd send it out, but the authors must rerun baselines under their own prompt protocol, report variance, and ideally evaluate with a different ASR. The prompt condition should be disclosed clearly and justified against the 'silent video' framing. This paper is for L2S specialists who will read it for the architectural ideas, but the WER numbers should not be cited as SOTA yet.","headline":"A genuinely new L2S architecture with a real evaluation leak: the ground-truth audio prompt and the Auto-AVSR circularity undermine the SOTA claim until re-baselined.","tokens_in":9710,"tokens_out":2784,"would_cite":false,"duration_ms":29486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RESOUND reconstructs intelligible, natural speech from silent talking-face video by decomposing generation into acoustic and semantic branches rooted in source–filter theory.","keywords":["lip-to-speech synthesis","source-filter theory","acoustic-semantic decomposition","speech units","prosody prediction","multimodal fusion","audio-visual speech recognition","speech synthesis"],"falsifier":"Run the paper's MOS protocol with human listeners on RESOUND's LRS3-TED outputs and decode the same outputs with a second, independently trained automatic speech recognizer; if the human ratings or the second recognizer's WER do not reproduce the advantage over LipVoicer and DiffV2S shown in Table 1, the intelligibility claim is partly an artifact of using the same recognizer for supervision and evaluation.","tokens_in":8686,"feed_emoji":"🎙️","tokens_out":9425,"duration_ms":86726,"temperature":0.7,"pith_summary":"Lip-to-speech synthesis must recover both what was said and how it was said from a silent face. RESOUND claims that these two jobs should be split, following source–filter theory: an acoustic branch predicts prosody — pitch, energy, timbre — from a short speaker prompt, while a semantic branch derives linguistic content from lip movements, lip-to-text predictions, and phonemes. The paper reports that on LRS3-TED this split lowers word error rate to 20.06 from 27.69 for the best compared baseline, raises naturalness to 3.002 UTMOS, and runs at near-real-time speed. If right, the work says that deliberately separating source and filter representations makes lip-to-speech easier to optimize and more practical.","feed_headline":"RESOUND cuts lip-to-speech WER to 20.06 on LRS3-TED","feed_subtitle":"Splitting prosody from content lifts naturalness to 3.002 UTMOS and keeps synthesis near real time.","key_machinery":"The load-bearing mechanism is a dual-path architecture inspired by source–filter theory. The Excitation Generator models the glottal source (pitch, energy, timbre, and a speaker prompt) while the Formant Generator models the vocal-tract filter (visual features and phonemes); their outputs are summed into a coarse mel-spectrogram that a Mel-Decoder refines, and a Linguistic Predictor emits discrete speech units that condition a vocoder alongside the mel-spectrogram. The semantic side uses AV-HuBERT to extract visual-linguistic features, a lip-to-text model whose output is converted by G2P into phoneme embeddings, and a Semantic Reference Attention module that aligns visual and phonetic sequences with a Reference Transformer followed by a Conformer. The prosody side uses a timbre encoder, a Speaker Prompt Encoder, and FastSpeech-2-style pitch and energy predictors, so each branch can be optimized independently.","core_discovery":"RESOUND's central claim is that decomposing speech generation into an excitation (acoustic/prosodic) representation and a formant (semantic/linguistic) representation, then fusing them additively into a coarse mel-spectrogram, yields state-of-the-art lip-to-speech quality. On LRS3-TED the system reports 20.06 WER, 3.002 UTMOS, 0.777 SECS, and 0.423 ESTOI, beating the compared systems on most metrics; on LRS2-BBC it improves most metrics except WER (28.55 versus 17.04 for LipVoicer), which the authors attribute to noise in that dataset. Ablations show that removing the lip-to-text branch and the Semantic Reference Attention raises WER by roughly 37%, and removing the acoustic branch degrades naturalness by about 28% and pitch error by about 40%. The acoustic branch also predicts pitch and energy explicitly, which the paper argues improves prosody beyond prior source-filter work that ignored energy.","pith_inferences":["Because the same recognizer (Auto-AVSR) both supplies the lip-to-text supervision and computes the reported WER, the cleanest test of the intelligibility claim is a human listening study or evaluation with a second, independently trained recognizer; the paper's numbers alone leave this overlap unsettled.","The explicit acoustic/semantic split suggests the model could edit prosody and content independently (for example, changing emotion while keeping the words), a capability the paper does not demonstrate.","Using a 0.5-second audio prompt for timbre together with visual content implies a possible path to voice cloning from silent video, with privacy implications the paper does not discuss.","The weaker LRS2 WER indicates the method's robustness in noisy, uncontrolled conditions is not yet established, so the general claim about real-world applicability should be read as strongest for relatively clean TED-style speech."],"forward_implications":["On LRS3-TED, RESOUND reaches 20.06 WER against 27.69 for the best compared baseline, with 3.002 UTMOS and 0.777 SECS; if the measurements hold, it is the best reported lip-to-speech result on that benchmark.","With RTF 0.063, it generates one second of speech in about 63 milliseconds and runs more than 200 times faster than LipVoicer (RTF 14.117), making real-time use plausible.","Ablations show that removing the lip-to-text branch and the Semantic Reference Attention raises WER by about 37%, while removing the acoustic branch lowers naturalness by about 28%; the two branches contribute independently.","On LRS2-BBC, the system improves most metrics but not WER (28.55 versus 17.04 for LipVoicer); the paper attributes the gap to noise in LRS2 and to LipVoicer trading naturalness for content accuracy.","When the vocoder is fed ground-truth mel-spectrograms and speech units, performance nearly closes the gap to ground truth (UTMOS 3.43, WER 1.21), indicating the encoding side, not the vocoder, carries most of the reconstruction burden."],"supporting_citations":[{"why":"Source–filter theory of speech production, the conceptual basis for splitting excitation from filter.","marker":"[15, 16]"},{"why":"Supplies the lip-to-text predictions used as semantic supervision and is the recognizer used for WER evaluation.","marker":"[22]"},{"why":"AV-HuBERT extracts frame-wise visual-linguistic features from the silent video.","marker":"[21]"},{"why":"G2P model converts the lip-to-text output into phoneme embeddings for the semantic branch.","marker":"[23]"},{"why":"FastSpeech 2 provides the pitch and energy predictor design and the phoneme encoder used in both branches.","marker":"[20]"},{"why":"FastPitchFormant is the prior neural source-filter TTS model that RESOUND extends to lip-to-speech.","marker":"[17]"},{"why":"Supplies speech units, the multi-input vocoder, and one of the main baselines.","marker":"[9]"},{"why":"DiffV2S defines the evaluation protocol (UTMOS, SECS, WER) and is the diffusion baseline.","marker":"[14]"},{"why":"LipVoicer is the strongest compared baseline on several metrics and the reference for the RTF comparison.","marker":"[12]"}],"fun_headline_variants":["RESOUND splits prosody and content to sharpen lip-to-speech","RESOUND reduces WER to 20.06 via acoustic-semantic split","RESOUND hits 20 WER and 3 UTMOS on lip-to-speech","RESOUND's source-filter design yields expressive silent-video speech","RESOUND: decomposing acoustic and semantic features improves speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the Auto-AVSR word-error rate is an unbiased measure of intelligibility even though the same Auto-AVSR model supplies the text predictions used as training supervision in the semantic branch.","fun_headline_variants_meta":{"raw":{"variants":["RESOUND splits prosody and content to sharpen lip-to-speech","RESOUND reduces WER to 20.06 via acoustic-semantic split","RESOUND hits 20 WER and 3 UTMOS on lip-to-speech","RESOUND's source-filter design yields expressive silent-video speech","RESOUND: decomposing acoustic and semantic features improves speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000892,"raw_usage":{"total_tokens":3832,"prompt_tokens":913,"completion_tokens":2919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2834}},"tokens_in":529,"tokens_out":2919,"duration_ms":23055,"temperature":1.0,"reasoning_tokens":2834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:16:09.190820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's MOS protocol with human listeners on RESOUND's LRS3-TED outputs and decode the same outputs with a second, independently trained automatic speech recognizer; if the human ratings or the second recognizer's WER do not reproduce the advantage over LipVoicer and DiffV2S shown in Table 1, the intelligibility claim is partly an artifact of using the same recognizer for supervision and evaluation.","supporting_citations":[{"cited_title":"Fant,Acoustic theory of speech production,","cited_arxiv_id":null,"evidence_quote":"Supplies the lip-to-text predictions used as semantic supervision and is the recognizer used for WER evaluation."},{"cited_title":"The source–filter theory of speech,","cited_arxiv_id":null,"evidence_quote":"AV-HuBERT extracts frame-wise visual-linguistic features from the silent video."},{"cited_title":"Lip-to-speech synthesis in the wild with multi-task learning,","cited_arxiv_id":null,"evidence_quote":"FastPitchFormant is the prior neural source-filter TTS model that RESOUND extends to lip-to-speech."},{"cited_title":"Lip to speech synthesis with visual context attentional gan,","cited_arxiv_id":null,"evidence_quote":"Supplies speech units, the multi-input vocoder, and one of the main baselines."},{"cited_title":"Revise: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,","cited_arxiv_id":null,"evidence_quote":"DiffV2S defines the evaluation protocol (UTMOS, SECS, WER) and is the diffusion baseline."},{"cited_title":"Lipvoicer: Generating speech from silent videos guided by lip reading,","cited_arxiv_id":null,"evidence_quote":"LipVoicer is the strongest compared baseline on several metrics and the reference for the RTF comparison."}],"review_version":1}