{"id":"548d5097-c60c-47a6-b4cd-0b2f7c0c31b2","arxiv_id":"2505.01263","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A dubbing system that combines an LLM, phoneme-level contrastive lip alignment, and flow matching with classifier-free guidance outperforms prior dubbing models on lip-sync and quality metrics.","lead":"FlowDubber generates movie dubbing speech that stays in sync with the speakers' lips while cloning a target voice. It combines a large language model with flow matching and contrastive learning, and reports large gains in lip-sync and quality scores on two benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation Table 5 shows removing DCA collapses LSE-C/D to baseline levels; the key open question is whether the SyncNet metric is capturing genuine synchronization or is distorted by the flow/LLM prior.","rationale":"The reader identifies the same concern, and the strongest piece of internal evidence against the metric's validity is the LSE-C on Chem 1.0 (8.21) exceeding ground truth (8.12), which I corroborate. My additional analysis of Table 5 shows that the entire reported LSE-C/D advantage is attributable to the DCA module, as removing it drops performance to roughly baseline levels. This sharpens the concern: the claim that DCA produces the observed SyncNet improvement is consistent, but whether that improvement translates to perceptual sync is untested. The paper has several independent supports: it is an accepted ACM MM 2025 paper, it reports ablations that isolate each module, and it follows a structurally reasonable design (LLM semantic learning, contrastive alignment, flow matching with guidance). There is no obvious internal inconsistency in the equations, and the contrastive losses in Eq. 7-9 are well-formed. The strongest weakness is in the evaluation layer, exactly at the point where the paper makes its headline claim. The reader's proposal to condition acceptance on human evaluation and artifact release is therefore the right move; the central claim can still be accepted if the human evaluation supports the large LSE-C/D gains, but as written the evidence is not conclusive. I agree with the reader's overall verdict and do not see a stronger concern in the method itself.","tokens_in":18474,"tokens_out":3200,"duration_ms":24314,"concrete_test":"Run a small human evaluation on the Chem 1.0 setting: N=15-20 raters with forced-choice A/B comparisons between FlowDubber output and ground-truth audio for lip-sync precision (visual-audio alignment), plus a MOS for clarity and speaker similarity. Also compute mean and per-sample error bars for SyncNet LSE-C/LSE-D over the test set, and report the correlation between per-sample LSE-C/D and human lip-sync ratings. If human sync judgments do not track the LSE-C/D advantage, or if the LSE-C/D advantage over GT disappears under human judgment, the central 'SOTA lip-sync' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central SOTA claim rests on the LSE-C/LSE-D numbers in Tables 1 and 3: FlowDubber achieves 8.21/6.89 on Chem 1.0 and 7.27/6.72 on GRID 1.0, versus best baselines of 3.87/10.92 and 6.12/9.03, respectively. This is the paper's load-bearing evidence because it is the only evidence that the proposed DCA/LLM alignment actually produces synchronization, and the scale of the improvement is the basis for 'SOTA'.\n\nThe paper's own Table 5, however, reveals what DCA does empirically: removing DCA (row 3) drops LSE-C from 8.21 to 3.62 and LSE-D from 6.89 to 10.28, to essentially the same range as the TTS-with-duration-predictor baselines (Table 1: 2.58-3.87 LSE-C, 10.56-12.54 LSE-D). This is exactly what one would expect if DCA is the component that converts the model from duration-predictor alignment to frame-level alignment, and it is also roughly what one would expect if DCA mainly causes the model to inject strong per-frame phoneme-conditioned energy into the mel prior.\n\nThe unresolved issue is whether SyncNet LSE-C/D is a valid proxy for the perceptual claim. The paper's Table 4 caption explicitly notes that DNSMOS, SNR, and UTMOS are not human subjective metrics, and Section 4.3 describes LSE-C/D as based on a SyncNet trained on in-the-wild data. Importantly, in Table 1, FlowDubber's LSE-C (8.21) exceeds ground-truth audio (8.12) on Chem. If the metric were measuring real synchronization quality, the generated audio should not systematically outscore the ground-truth audio that is the source of the visual signal. This specific observation is the clearest sign that the metric may be rewarding something other than perceptual synchronization. The absence of any human evaluation (MOS for subjective quality, or human-judged lip-sync precision) means the SOTA claim is only as strong as the metric's validity in this setting.\n\nAdditionally, the guidance scale alpha in Eq.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlowDubber, an end-to-end movie dubbing system combining a Qwen2.5-based speech language model with semantic-aware phoneme learning, dual contrastive alignment (DCA) between lip motion and phonemes, and a flow-matching decoder enhanced by classifier-free guidance. The authors report large improvements over three prior dubbing methods on the Chem and GRID benchmarks, particularly in SyncNet-based lip-sync metrics (LSE-C/LSE-D), with comparable or slightly better WER and UTMOS. The ablations show that removing DCA collapses lip-sync performance, removing LLM-SL degrades WER significantly, and the proposed FVE guidance raises automatic clarity scores. The paper claims state-of-the-art results on both benchmarks.","tokens_in":18823,"tokens_out":2932,"duration_ms":31213,"significance":"If the results are valid, the paper makes a strong contribution: it demonstrates that a lip-motion-conditioned alignment module can be integrated into an LLM-based speech synthesis pipeline, and that the resulting system can achieve near-ground-truth automatic lip-sync scores while cloning a reference voice. The architectural combination is innovative and internally consistent, and the ablations are informative: Table 5 clearly attributes the lip-sync gain to DCA and the pronunciation gain to LLM-SL. The paper also ships a demo page and uses reproducible public benchmarks. The main qualification is epistemic: all headline metrics are automatic predictors, no human evaluation is reported, and some recent dubbing baselines cited in the related work are absent from the comparisons. These issues affect the strength of the central SOTA claim but do not invalidate the architecture itself.","major_comments":[{"comment":"The central claim that FlowDubber 'sets SOTA results on both Chem and GRID benchmarks' is based only on comparisons with StyleDubber, Speaker2Dubber, and ProDubber. Several more recent dubbing methods cited in Section 2.1 and the reference list, including EmoDubber [12], MCDubber [83], VoiceCraft-Dub [56], DeepDubber-V1 [84], and FCConDubber [36], are not evaluated in any table. To support the SOTA claim, at least the strongest of these recent methods should be included in the benchmark comparison, or the claim should be qualified to the three compared baselines.","section":"Tables 1-3"},{"comment":"The paper's headline claims of 'high-quality audio-visual sync' and 'better acoustic quality' rest entirely on automatic predictors: LSE-C/LSE-D from SyncNet, UTMOS, DNSMOS, SNR, SIM-O, and WER. The Table 4 caption itself states that DNSMOS, SNR, and UTMOS are not human subjective metrics, but no human listening or subjective MOS evaluation is reported anywhere in the paper. This is a load-bearing gap because the lip-sync claim is the main differentiator of the method. A human perceptual evaluation, or at least an analysis of how well the automatic metrics correlate with human judgments on these datasets, is needed before the SOTA claim can be accepted.","section":"Section 4.3 and Table 4 caption"},{"comment":"FlowDubber's LSE-C on Chem (8.21) exceeds the ground-truth audio's LSE-C (8.12), and its LSE-D (6.89) is close to ground truth (6.59). If LSE-C/D measure genuine synchronization with the video, a generated utterance should not systematically outperform the real recorded audio. This anomaly suggests that SyncNet may reward artifacts of the generated mel-spectrogram or that the metric is not a faithful proxy for perceptual sync quality. The paper should explain this behavior or provide evidence, such as per-sample analysis or a human sync evaluation, that the generated audio is truly as well synchronized as the ground truth.","section":"Table 1, Setting 1.0"},{"comment":"The guidance scale alpha is tuned empirically to maximize DNSMOS, SNR, and UTMOS on the same test set, and the same three metrics are then used to claim that FVE improves clarity. This is circular: the monotonic trend in Table 4 is expected when alpha is selected to optimize those exact predictors. The paper should report alpha selection on a validation set and evaluate the chosen model with held-out metrics or human listening, otherwise the clarity improvement claim is not independently established.","section":"Section 4.5 and Table 4"},{"comment":"All reported numbers are single-run point estimates without error bars, confidence intervals, or statistical significance tests. Given that the performance gaps in Tables 1 and 3 are extremely large for LSE-C/D, a small number of runs across multiple seeds would materially strengthen the claim and would help rule out metric instability or training stochasticity as an explanation.","section":"Tables 1-6"}],"minor_comments":[{"comment":"The header 'UTMOS↓' is incorrect: higher UTMOS is better, as stated in Table 1. The arrow should be 'UTMOS↑'.","section":"Table 5"},{"comment":"The formula 'mu = F([C_lip, Up(LLM_p, z_p, t_ab))' has mismatched parentheses; it should be 'F([C_lip, Up(LLM_p, z_p, t_ab)])'.","section":"Eq. (10)"},{"comment":"The second line of Eq. (4) appears to have duplicated arguments: 'f_theta(LN(Z_hat) + LN(Z_hat))' should likely be 'LN(Z_hat) + LN(Z_prev)' or similar. Please clarify.","section":"Eq. (4)"},{"comment":"The text says 'absolute improvements of 5.63% and 5.65%' for LSE-C and LSE-D, but these appear to be absolute differences in metric values, not percentages. Consider rewording to 'improvements of 5.63 and 5.65 in LSE-C and LSE-D'.","section":"Section 4.4, Chem results"},{"comment":"The ablation of audio generators is useful, but the statement that for DAC and CV the original waveform is generated and then reconstructed is unclear. Clarify whether the flow-matching mel-spectrogram is decoded, re-encoded, and then vocoded, or whether the codec is used in some other reconstruction pipeline.","section":"Section 4.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for ACM MM and the architecture is interesting, but the SOTA claim currently depends on automatic metrics, an alpha-selection procedure that is circular with the evaluation, and a baseline set that excludes several recent dubbing methods. I could support acceptance after the authors add a human evaluation, include the stronger recent baselines, and address the LSE-C-above-ground-truth anomaly. I do not see the issues as irreparable, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nFlowDubber is a serious dubbing paper. The new combination is real: Qwen2.5 speech LLM, phoneme-level semantic features, dual contrastive lip-phoneme alignment, and flow matching with decoupled CFG. The ablations show every component earns its place, and the DCA module in particular is doing heavy lifting — removing it collapses LSE-C/D to baseline levels. That is what a real alignment mechanism should look like, not a cosmetic add-on. The zero-shot test is also a good addition.\n\nThe soft spots are mostly in the evaluation, not the architecture. The entire SOTA claim rests on SyncNet LSE-C/D, UTMOS, DNSMOS, and SNR. The paper itself admits DNSMOS, SNR, and UTMOS are not human subjective metrics, and there is no human listening or lip-sync judgment anywhere. Worse, FlowDubber's LSE-C on Chem (8.21) beats ground truth (8.12). That is a warning sign: either the metric is not a faithful proxy for perceived synchronization, or the model is injecting artifacts that SyncNet reads as confidence. The guidance scale alpha is also tuned empirically on those same predictors in Table 4, then used to report final numbers, so the clarity gain is partially circular. And the headline comparisons omit several cited recent dubbing models — VoiceCraft-Dub, FCConDubber, MCDubber, EmoDubber — which weakens the 'SOTA' phrasing.\n\nNone of this kills the paper. The core idea is coherent and the experimental design is otherwise reasonable. But the magnitude of the claimed advance — near-ground-truth lip-sync — is not yet established. It needs human evaluation, ideally a forced-choice sync test and a naturalness MOS, plus error bars or released code so others can check whether the metric advantage survives.\n\nWho is this for? Anyone working on video dubbing, audio-visual speech synthesis, or flow-matching TTS. It will be a useful reference even if the metric story softens. I would accept it for peer review — it is a substantive systems paper with credible ablations — but the acceptance should be conditional on addressing the evaluation gap, not desk-rejected.","headline":"Plausible dubbing architecture with a real lip-sync contribution, but the headline numbers rest on automatic predictors that may reward artifacts; worth refereeing with a demand for human evaluation.","tokens_in":19504,"tokens_out":2064,"would_cite":true,"duration_ms":21287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlowDubber claims that replacing duration prediction with LLM-based semantic-aware learning, dual contrastive lip-phoneme alignment, and flow-matching voice enhancement yields near-ground-truth lip-sync and competitive word error rates on…","keywords":["movie dubbing","visual voice cloning","lip synchronization","flow matching","speech language model","contrastive alignment","voice cloning","classifier-free guidance"],"falsifier":"A forced-choice human study on held-out Chem and GRID clips would settle the perceptual claim: if raters do not consistently prefer FlowDubber's dubs to the strongest prior system for lip-sync and clarity, or if the guidance-scale setting that maximizes the neural clarity scores is not the one humans prefer, the reported metric gap is not evidence of perceptual gain. A cheaper diagnostic is to check whether the lip-sync metric still rises when synthetic sharpening is added at the wrong moments; if it does, the metric is gameable.","tokens_in":18253,"feed_emoji":"🎬","tokens_out":9896,"duration_ms":90182,"temperature":0.7,"pith_summary":"FlowDubber sets out to fix two weaknesses it attributes to earlier movie-dubbing systems: coarse lip-sync, because duration prediction only approximates timing, and degraded acoustic quality, because standard vocoder pipelines blur spectral detail. The paper's solution is an architecture that replaces the duration predictor with a speech language model that learns phoneme-level semantics from the script and reference voice, a dual contrastive alignment that ties those phonemes to lip motion in both directions, and a flow-matching voice-enhancement stage that sharpens clarity while preserving speaker identity. The headline result is that on the Chem and GRID dubbing benchmarks this architecture reaches lip-sync scores near the ground-truth audio: on Chem setting 1.0, lip-sync confidence (LSE-C) is 8.21 versus ground truth 8.12 and the best baseline 3.87, while word error rate stays comparable to the previous state of the art. If true, the payoff is that dubbing could achieve near-ground-truth automatic lip-sync and voice cloning from a short reference clip without a separate duration-prediction stage.","feed_headline":"Dubbing nears ground-truth lip-sync with LLM flow matching","feed_subtitle":"The FlowDubber pipeline replaces duration prediction with phoneme-lip alignment and flow-based voice enhancement.","key_machinery":"Three modules carry the argument. LLM-based Semantic-aware Learning (LLM-SL) first converts the reference audio into semantic and global tokens through speech tokenization, then feeds script text, semantic tokens, and global tokens to a pretrained text-based language model that autoregressively predicts speech tokens, and finally adapts those token semantics to the phoneme level with cross-modal transformers whose keys and values come from the LLM-mapped speech features. Dual Contrastive Aligning (DCA) takes phoneme embeddings and lip-motion embeddings extracted from the silent video, applies contrastive losses in both directions so each lip frame retrieves its temporally aligned phoneme and vice versa, builds a similarity matrix from the learned alignment, and uses monotonic alignment search to convert that matrix into a per-phoneme frame mapping. Flow-based Voice Enhancing (FVE) then trains a conditional flow-matching network with an optimal-transport interpolation from noise to mel-spectrogram, injects speaker style through an affine prior, and refines the generated spectrum with a decoupled classifier-free guidance term that amplifies the LLM-conditioned vector field over a zero-semantics field. Together these convert script, reference voice, and silent video into a mel-spectrogram prior that already carries the lip alignment before any vocoder is applied.","core_discovery":"The paper claims that an LLM-based semantic-aware learning module plus dual contrastive alignment makes lip-sync a first-class objective rather than a byproduct of duration prediction, and that a flow-matching decoder with LLM-based guidance produces clearer audio than prior dubbing vocoders. Concretely, FlowDubber reports state-of-the-art results on both benchmark datasets. On Chem setting 1.0 it reports LSE-C 8.21 (ground truth 8.12, best prior dubbing baseline 3.87), LSE-D 6.89 (ground truth 6.59, best prior 10.56), UTMOS 3.91, SIM-O 0.754, and WER 9.96; on GRID it reaches LSE-C 7.27 (ground truth 7.13) at setting 1.0. In a zero-shot test with an unseen reference speaker it beats all compared methods on every reported metric. The ablations attribute pronunciation to the LLM semantic learning, lip-sync to the dual contrastive alignment, and identity to the style prior in flow matching.","pith_inferences":["The reported LSE-C exceeding ground-truth audio's LSE-C on Chem (8.21 vs 8.12) suggests the lip-sync metric may reward artifacts; a natural extension is to test whether human raters still prefer FlowDubber over real audio.","The guidance scale alpha is presented only through clarity metrics; testing whether higher alpha changes LSE-C/LSE-D or speaker similarity would map the trade-off between the LLM-enhancement path and the visual alignment path.","The decoupling of LLM semantics from lip-aligning features suggests the method could be ported to cross-lingual dubbing by keeping the alignment path and swapping the text-phoneme side, but the paper does not test this.","Since all acoustic-quality conclusions rest on learned predictors, an independent human-subject study is the cheapest way to decide whether the voice-enhancing contribution is perceptual or only numeric."],"forward_implications":["On the paper's evidence, a movie-dubbing system can reach near-ground-truth lip-sync on automatic metrics without a duration predictor, by learning mutual alignment between phoneme and lip-motion embeddings.","The LLM's phoneme-level semantic features are what keep pronunciation accurate: removing LLM-SL in the ablation raises WER from 9.96 to 48.33 on Chem setting 1.0.","The division of labor is clean: the flow-matching style prior carries speaker identity, and the LLM-based guidance scale monotonically improves clarity metrics (DNSMOS, SNR, UTMOS) when increased from 0.0 to 0.8.","Because lip alignment is baked into the mel-spectrogram prior before the vocoder, the same front-end can work with different neural vocoders and still preserve synchronization; the paper shows this by swapping in alternative generators and retaining high LSE-C/LSE-D.","The system generalizes to unseen reference voices in a zero-shot setting, so it does not require per-speaker adaptation to keep its lip-sync advantage."],"supporting_citations":[{"why":"defines the Visual Voice Cloning task and sets the script-plus-reference-audio formulation that FlowDubber adopts.","marker":"[4]"},{"why":"supplies the lip-sync confidence and distance metrics that carry the paper's headline comparison.","marker":"[10]"},{"why":"provides the lip-motion feature extractor and attention-based alignment that DCA builds on and compares against.","marker":"[11]"},{"why":"is the duration-predictor baseline whose coarse timing FlowDubber claims to surpass in lip-sync.","marker":"[13]"},{"why":"provides flow matching, the generation backbone of the voice-enhancing module.","marker":"[38]"},{"why":"supplies forced-alignment phoneme timings used to define positive lip-phoneme pairs in the contrastive losses.","marker":"[44]"},{"why":"supplies the UTMOS acoustic-quality metric used to compare synthesis quality.","marker":"[54]"},{"why":"supplies the speech tokenization and speech-language-model design that LLM-SL is built on.","marker":"[64]"},{"why":"is the strongest previous dubbing baseline whose WER and quality numbers FlowDubber must match or beat.","marker":"[81]"}],"fun_headline_variants":["LLM flow matching dubbing improves lip-sync and clarity","FlowDubber: LLM-driven dubbing hits new lip-sync scores","Dubbing gets clearer audio and lip-sync via flow matching","LLM-based dubbing model tops benchmarks in lip-sync, clarity","FlowDubber uses LLM and flow for better dubbing audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated scores used as headline evidence—a learned lip-sync confidence/distance measure and neural estimates of clarity and naturalness—faithfully capture what human viewers and listeners perceive, since the paper reports no human evaluation and on one setting its lip-sync score exceeds the ground-truth audio's score.","fun_headline_variants_meta":{"raw":{"variants":["LLM flow matching dubbing improves lip-sync and clarity","FlowDubber: LLM-driven dubbing hits new lip-sync scores","Dubbing gets clearer audio and lip-sync via flow matching","LLM-based dubbing model tops benchmarks in lip-sync, clarity","FlowDubber uses LLM and flow for better dubbing audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2779,"prompt_tokens":1025,"completion_tokens":1754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1662}},"tokens_in":641,"tokens_out":1754,"duration_ms":12875,"temperature":1.0,"reasoning_tokens":1662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:22:40.462066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A forced-choice human study on held-out Chem and GRID clips would settle the perceptual claim: if raters do not consistently prefer FlowDubber's dubs to the strongest prior system for lip-sync and clarity, or if the guidance-scale setting that maximizes the neural clarity scores is not the one humans prefer, the reported metric gap is not evidence of perceptual gain. A cheaper diagnostic is to check whether the lip-sync metric still rises when synthetic sharpening is added at the wrong moments; if it does, the metric is gameable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies forced-alignment phoneme timings used to define positive lip-phoneme pairs in the contrastive losses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the strongest previous dubbing baseline whose WER and quality numbers FlowDubber must match or beat."}],"review_version":1}