{"id":"7adf80f8-f53b-4f30-bb72-c0320046bd78","arxiv_id":"2412.16530","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding a SyncNet-based lip-synchrony loss plus a duration loss to a pre-trained audio-visual speech-to-speech model improves lip-sync of overlaid translated audio on original videos across four language pairs.","lead":"This paper adds a lip-synchrony loss to the training of a direct audio-visual speech-to-speech translation system, fine-tuning only the duration predictor while leaving the translation model frozen. On four language pairs, the method cuts lip-sync distance by 9.2 percent on average versus a strong baseline without lowering translation quality or audio naturalness scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No duration-loss-only control and shared SyncNet train/eval metric leave the sync-loss attribution unproven.","rationale":"The paper's claimed contribution is specifically that adding a SyncNet lip-sync loss to duration-predictor fine-tuning improves lip-synchrony without quality loss. For that claim to be load-bearing, Lsync must do work beyond Ldur, and the LSE-D metric must be a trustworthy, non-circular measure of lip-sync. Section 6.1's ablation matrix omits the D-only cell, so the best case (LS+D FT) could differ from a plain duration-matching fine-tune only by chance or by the SyncNet objective. The absence of error bars or significance detail (despite a claimed p<0.05) makes it impossible to tell whether the 11.7-to-10.7 gap in En-Es is real. The shared SyncNet in Eq. 2 and Section 4.2 is an additional confound: optimizing the evaluator directly is known to inflate scores, and no independent metric or human judgment is reported. I do not think this refutes the empirical result; the improvements are modest and the paper honestly shows content-level changes achieve more at translation cost. But the central attribution is not yet established. A single D-only ablation would largely resolve it, and an independent sync evaluation would rule out metric overfitting. Thus the conditional verdict should stand unchanged, with those checks as conditions.","tokens_in":8521,"tokens_out":8487,"duration_ms":71168,"concrete_test":"Run the missing ablation: fine-tune the duration predictor with Eq. 1 only, setting λ=10 and Lsync=0, from the same AV2AV pretrained checkpoint, for the same 200K iterations, on En-Es and ideally all four language pairs. Compare LSE-D/LSE-C to LS+D FT in Table 4. If D-only reaches approximately 10.7 LSE-D (within 0.2 of LS+D FT), the sync loss contributes essentially nothing and the paper's central attribution fails; if D-only remains near 11.7, the sync loss is the active ingredient. As a secondary check, score all outputs with an independently trained lip-sync model or a human perceptual test to rule out metric overfitting from the shared SyncNet objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that adding the SyncNet lip-sync loss Lsync (Eq. 2) to duration-predictor fine-tuning causes the LSE-D improvement from 11.75 (AV2AV-Speech average) to 10.67. For this attribution to hold, two conditions must be met: (i) Lsync must contribute beyond the ordinary duration loss Ldur (Eq. 1), and (ii) LSE-D must measure lip-synchrony independently of the optimized objective. Neither is currently supported. Section 6.1 (Table 4) ablates LS+D, LS+D-from-scratch, and LS-only, but never D-only. Since the duration loss already forces generated speech to match source speech duration, it may by itself reduce LSE-D by tightening temporal alignment; the missing D-only row means the improvement credited to LS+D FT could come entirely from Ldur. Compounding this, Section 4.2 evaluates with LSE-C/LSE-D from Wav2Lip [7], which use the same SyncNet [27] expert that appears in Eq. 2 as Lsync. The reported gains are therefore partly a train/test overlap: the model directly optimizes the network that produces the headline metric. The claim of p-value<0.05 is not accompanied by number of samples, variance, or test procedure. These gaps do not prove the result false, but they leave the causal story that the lip-sync loss improves lip-synchrony unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses lip-synchrony in direct audio-visual speech-to-speech translation (AVS2S) by fine-tuning the duration predictor of an existing AV2AV model with two auxiliary losses: a standard duration loss (Eq. 1) and a SyncNet-based lip-synchrony loss (Eq. 2). The unit-to-unit translation decoder and vocoder remain frozen, and the generated speech is overlaid on the original video without modifying the visual content. Experiments on LRS3 across English-to-Spanish, Portuguese, Italian, and French show that the proposed method reduces the average LSE-D from 11.75 to 10.67 relative to the AV2AV-Speech baseline, with roughly unchanged PESQ, BLASER, and ASR-BLEU. Ablations and an additional paraphrase-based translation-model experiment are reported.","tokens_in":8754,"tokens_out":4663,"duration_ms":37551,"significance":"If the reported results are robust, the paper offers a practical, lightweight way to improve lip-sync in dubbing without face synthesis, thereby avoiding deepfake-related ethical concerns. The work is clearly motivated, builds on an open-source strong baseline, and evaluates on four language pairs with consistent trends. The main caveats are that the attribution of the improvement to the sync loss is not yet established (missing D-only control) and that the evaluation metric and training loss share the same SyncNet expert. The paper also provides a useful negative result in Section 6.2 showing that content-level changes improve synchrony further at the cost of translation quality.","major_comments":[{"comment":"The ablation table compares LS+D FT, LS+D from scratch, and LS FT, but it never includes a duration-loss-only (D-only) condition. Since the duration loss in Eq. 1 already forces the generated speech to match source durations, it may by itself reduce LSE-D by improving temporal alignment. Without a D-only row, the improvement attributed to the sync loss (LS) in the headline result (Table 2) is not identified, because the combined LS+D FT result could be driven entirely by Ldur. Please add a D-only fine-tuning run from the same pretrained checkpoint and report its LSE-C/LSE-D.","section":"Section 6.1, Table 4"},{"comment":"The evaluation metrics LSE-C and LSE-D are computed with SyncNet [27], which is exactly the network that provides the training loss Lsync in Eq. (2). Because Lsync is defined in terms of SyncNet confidence, the fine-tuning procedure directly optimizes the network that produces the headline evaluation metric. The reported gains may therefore partly reflect train/test overlap rather than a genuinely better audio-visual alignment. The authors should either add an evaluation with a loss-agnostic lip-sync metric (e.g., a different pretrained sync detector or human ratings) or show that the improvement persists when Lsync is replaced by a different objective; at minimum, a model trained with Ldur only should be compared to quantify the overlap.","section":"Section 4.2 and Eq. (2)"},{"comment":"The statement \"Ours significantly outperform both Synthetic and AV2AV-Speech approaches in terms of lip-synchrony scores (p-value<0.05)\" is not accompanied by any details of the statistical test: number of samples (videos or utterances), variance of LSE-D across samples, test type (paired or unpaired), or the value of the test statistic. Given that Table 2 shows per-language LSE-D differences of about 0.7-1.9 and that the baseline itself varies by language, a paired test with confidence intervals on the mean difference is needed to substantiate the claim. Please provide the full statistical details.","section":"Section 5.2"}],"minor_comments":[{"comment":"The arrows in the headers of Tables 5 and 6 are inconsistent with the definitions in Table 2: LSE-D is better when lower (should be ↓) and LSE-C is better when higher (should be ↑). The current notation ('LSE-D (↑)' and 'LSE-C (↓)' in Table 5) is reversed and will confuse readers.","section":"Tables 5 and 6"},{"comment":"Please define the units of d and d_p (e.g., predicted log-duration from the length predictor) and clarify whether the 'target duration' d is the source speech duration or a derived alignment target. This will make the loss and the fine-tuning procedure reproducible.","section":"Eq. (1)"},{"comment":"References [7] and [21] are the same work (Wav2Lip) and should be consolidated; the duplication is visible in the reference list.","section":"References"},{"comment":"The claim of 'no degradation' in PESQ, BLASER, and ASR-BLEU is based on very small numeric differences (e.g., BLASER differences of 0.001-0.003). Reporting these without variance or significance testing is acceptable as a descriptive statement, but please soften the claim to 'no substantial degradation' or add supporting intervals.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a clear ethical motivation, but the causal attribution issue is significant. I would require the D-only ablation and an independent metric before accepting. The authors should also consider releasing the fine-tuned duration predictor for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a straightforward empirical paper from Amazon on improving lip-sync in direct audio-visual speech-to-speech translation without generating video. They fine-tune the duration predictor of the AV2AV model with a SyncNet-based lip-sync loss plus a duration loss, while keeping the video untouched. The headline result is a 9.2% LSE-D reduction (from ~11.75 to 10.67 average over four Romance targets) with no drop in PESQ, BLASER, or ASR-BLEU. That result is consistent across language pairs, and the method is simple and practical. The paper also reports a useful negative finding: retraining the unit-to-unit decoder on paraphrases improves lip-sync further but degrades translation quality, so they abandon it.\n\nWhat's genuinely new is applying the Wav2Lip-style sync loss to the duration predictor of a direct AVS2S system, rather than to a video generator. That is an extension of prior work, not a conceptual leap, but the empirical validation is the contribution. The ablations show that combining both losses works better than the sync loss alone, and that pretrained initialization helps.\n\nThe soft spots are real. Table 4 never runs the duration loss alone, so we cannot tell whether the sync loss adds anything beyond what the duration loss already does. Given that the duration loss explicitly forces generated speech to match source duration, it may be the main driver of the LSE-D gain. Also, the training loss and the evaluation metric both use the same SyncNet model, so part of the reported gain could be overfitting to that network. A third issue: the p<0.05 claim is not backed by any significance-test details.\n\nThese are addressable and don't refute the practical claim that their method improves lip-sync over a strong baseline. But the causal story—that the lip-sync loss is responsible—is not yet proven. A reviewer should ask for a D-only ablation and an evaluation with an independent sync measure, or at least confidence intervals.\n\nI'd send this to peer review. It's a solid, honest piece of engineering for the AVS2S/dubbing community. The gaps are fixable in revision, and the motivation around preserving original video is sensible.","headline":"A practical lip-sync improvement for AVS2S, but the missing duration-loss-only ablation and shared SyncNet train/eval metric leave the sync-loss attribution unproven.","tokens_in":9346,"tokens_out":2631,"would_cite":false,"duration_ms":22648,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a SyncNet-based lip-synchrony loss to the duration predictor of a direct audio-visual speech-to-speech translation model improves dubbed lip-sync by 9.2% while leaving translation quality and naturalness unchanged.","keywords":["audio-visual speech-to-speech translation","lip-synchrony","SyncNet","duration prediction","dubbing","discrete speech units","lip-sync error distance","speech-to-speech translation"],"falsifier":"Run the same fine-tuning on a language pair whose target speech units cannot reproduce the source language's visible mouth-shape distinctions, and measure LSE-D on held-out videos; if the score does not improve over the untuned baseline, the timing-only assumption is the limiting factor and the reported gains depend on content-level choices.","tokens_in":8320,"feed_emoji":"🎬","tokens_out":13284,"duration_ms":111895,"temperature":0.7,"pith_summary":"The paper tackles lip-synchrony in direct audio-visual speech-to-speech translation (AVS2S): when translated speech is overlaid on the original video, existing models have no training signal that ties the generated audio to the speaker's visible mouth movements. The authors add a lip-synchrony loss, computed by a pretrained network that scores audio-video synchrony, together with a duration loss, and apply both losses only while fine-tuning the duration predictor; the unit-to-unit translation decoder stays frozen. Across English-to-Spanish, English-to-Portuguese, English-to-Italian, and English-to-French, this lowers the average lip-sync distance LSE-D from 11.75 to 10.67, a 9.2% improvement, with no measured degradation in speech quality or translation quality. The importance is that dubbed videos can become more realistic without generating or modifying faces, sidestepping visual artifacts and the ethical concerns around recreating a speaker's likeness.","feed_headline":"Lip-sync loss cuts dubbing mismatch by 9.2%","feed_subtitle":"Fine-tuning only the duration predictor keeps translation quality and naturalness intact across four language pairs.","key_machinery":"The load-bearing component is the duration predictor, a network that decides how long each discrete target speech unit should be held before the vocoder turns it into audio. The paper fine-tunes only this module with a combined loss made of a sync loss and a duration loss. The sync loss uses SyncNet, a pretrained audio-visual synchrony scorer, to penalize generated audio whose temporal alignment with the original video's lip movements is poor; the duration loss keeps predicted unit durations close to the source speech duration. Since the unit-to-unit decoder is frozen, the sync loss can only act through timing, stretching or compressing units rather than changing the content of the translation. That frozen-decoder constraint is exactly what makes the method lightweight, and also what sets its ceiling.","core_discovery":"On the paper's own terms, the discovery is that lip-synchrony in a direct AVS2S system can be improved by fine-tuning only the duration predictor with a combined sync and duration loss, while leaving the unit-to-unit decoder frozen. The predictor learns to stretch or compress each target speech unit so that the generated audio lands on moments where the source video's mouth shape is compatible, and the synchrony scorer supplies the gradient that points toward those moments. The method reaches an average lip-sync distance (LSE-D, lower is better) of 10.67, a 9.2% reduction from the speech-overlay baseline, and the improvement is consistent in every language direction. Automated measures of audio quality (PESQ) and translation quality (BLASER-2.0 and ASR-BLEU) stay essentially flat, so the sync gain is not bought by degrading the translation. Ablations show both losses are needed and that starting from the pretrained checkpoint matters.","pith_inferences":["An implication beyond the paper's claims is that the method's ceiling is set by the frozen decoder: if the available target speech units cannot approximate the source speaker's visible mouth shapes, retiming alone will saturate, and further sync gains would require content-level changes.","The paper's paraphrase experiment hints that content-level choice is a stronger lever on lip-sync than timing, but the authors find that unconstrained paraphrase generation lowers translation quality; constraining paraphrase generation to preserve meaning while matching mouth shapes is a natural next step.","The same duration-predictor fine-tuning recipe could transfer to any unit-based speech synthesizer that must align with a reference video, independent of the translation component.","A human perceptual study of dubbing would be a stronger test of practical benefit than the geometric LSE-D metric, which measures embedding distance rather than perceived naturalness."],"forward_implications":["Dubbed videos that keep the original footage can become more realistic without any face synthesis, avoiding visual artifacts and the identity/likeness risks of regenerated mouths.","The sync gain is consistent across four target languages, so the timing-based loss transfers without language-specific tuning.","Because only the duration predictor is fine-tuned, the method can be applied on top of an existing frozen unit-based translation pipeline with a modest training step.","The reported metrics show the lip-sync improvement does not come at the expense of speech naturalness or translation fidelity."],"supporting_citations":[{"why":"Supplies the AVS2S architecture, the pretrained weights that are fine-tuned, and the speech-overlay baseline that the method is compared against.","marker":"[6]"},{"why":"Contributes the SyncNet-based lip-sync loss formulation and the LSE-C/LSE-D metrics used for evaluation.","marker":"[7]"},{"why":"Provides the SyncNet model used as the audio-visual synchrony expert in the sync loss.","marker":"[27]"},{"why":"Supplies the large talking-head video dataset used to fine-tune the duration predictor and to evaluate on a disjoint test set.","marker":"[28]"},{"why":"Provides the pretrained audio-visual encoder that converts input video and speech into discrete unified units.","marker":"[23]"},{"why":"Defines the unit-to-unit translation module that maps source units to target units and is kept frozen during fine-tuning.","marker":"[24]"},{"why":"The vocoder that renders target units into speech, enabling full audio generation at each fine-tuning step for sync evaluation.","marker":"[25]"}],"fun_headline_variants":["Lip-sync loss improves dubbing by 9.2%","Fine-tuning duration predictor boosts lip-sync 9.2%","New sync loss tightens lip-sync in A-V translation","AVS2S lip-sync improved 9.2% with focused training","Sync-aware training cuts lip mismatch in translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that getting the timing of the target speech units right is enough to make the lips look synchronized, because the unit-to-unit decoder is frozen and cannot choose different sounds that would better match the speaker's visible mouth movements.","fun_headline_variants_meta":{"raw":{"variants":["Lip-sync loss improves dubbing by 9.2%","Fine-tuning duration predictor boosts lip-sync 9.2%","New sync loss tightens lip-sync in A-V translation","AVS2S lip-sync improved 9.2% with focused training","Sync-aware training cuts lip mismatch in translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2337,"prompt_tokens":902,"completion_tokens":1435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1357}},"tokens_in":518,"tokens_out":1435,"duration_ms":10033,"temperature":1.0,"reasoning_tokens":1357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:29:48.112038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning on a language pair whose target speech units cannot reproduce the source language's visible mouth-shape distinctions, and measure LSE-D on held-out videos; if the score does not improve over the untuned baseline, the timing-only assumption is the limiting factor and the reported gains depend on content-level choices.","supporting_citations":[{"cited_title":"not trading off lip-synchrony improvements over speech translation quality and naturalness","cited_arxiv_id":null,"evidence_quote":"Supplies the AVS2S architecture, the pretrained weights that are fine-tuned, and the speech-overlay baseline that the method is compared against."},{"cited_title":"Our A VS2S framework incorporates lip-synchrony and duration loss to enhance the alignment between speech and lip movements in audio-visual translation models","cited_arxiv_id":null,"evidence_quote":"Contributes the SyncNet-based lip-sync loss formulation and the LSE-C/LSE-D metrics used for evaluation."},{"cited_title":"Jointly optimizing translations and speech timing to improve isochrony in automatic dubbing,","cited_arxiv_id":null,"evidence_quote":"Provides the SyncNet model used as the audio-visual synchrony expert in the sync loss."},{"cited_title":"A lip sync expert is all you need for speech to lip generation in the wild,","cited_arxiv_id":null,"evidence_quote":"Supplies the large talking-head video dataset used to fine-tune the duration predictor and to evaluate on a disjoint test set."},{"cited_title":"Isometric MT: Neural Machine Translation for Automatic Dubbing","cited_arxiv_id":"2112.08682","evidence_quote":"Provides the pretrained audio-visual encoder that converts input video and speech into discrete unified units."},{"cited_title":"Duration modeling of neural tts for au- tomatic dubbing,","cited_arxiv_id":null,"evidence_quote":"Defines the unit-to-unit translation module that maps source units to target units and is kept frozen during fine-tuning."},{"cited_title":"Prosodic alignment for off-screen automatic dubbing,","cited_arxiv_id":null,"evidence_quote":"The vocoder that renders target units into speech, enabling full audio generation at each fine-tuning step for sync evaluation."}],"review_version":1}