{"id":"412f3a7b-2553-4fbc-bf70-90d9090483a4","arxiv_id":"2505.16279","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A movie dubbing system that uses a vision-language model to extract scene type and speaker attributes from silent video, then feeds those attributes as extra conditions into a diffusion-based speech generator, supported by a new 7.2-hour annotated movie dataset.","lead":"MM-MovieDubber adds a vision-language model to movie dubbing, reading the silent video to classify each line as dialogue, narration, or monologue and to estimate speaker age, gender, and emotion before generating speech. The authors report better speaker similarity, emotion similarity, and word accuracy than several dubbing baselines, and they introduce a new 7.2-hour annotated movie dubbing dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is confounded: the F5-TTS baseline appears to be off-the-shelf rather than fine-tuned on the dubbing datasets, so the reported gains may come from domain fine-tuning rather than the VLM/ControlNet conditions.","rationale":"The reader's weakest-assumption analysis identifies exactly the confound I would stress-test: the proposed model is fine-tuned on the target dubbing datasets while the F5-TTS baseline appears to be pretrained only, making the comparison unfair and the attribution of gains to the multi-modal components unproven. This is a genuine methodological concern, not a disagreement with consensus. It is load-bearing because the paper's headline 'superior performance compared to SOTA' and its novelty narrative both depend on the multi-modal conditions being the cause of the improvements. The paper itself provides partial evidence relevant to this concern: the ablation in Table 6 shows that removing the video understanding conclusion causes only small drops (SPK-SIM from 82.99 to 82.59, EMO-SIM from 64.74 to 63.18), and on GRID the proposed model is 'competitive' rather than uniformly superior on WER. These observations do not refute the multi-modal contribution, but they leave open the possibility that most of the gain comes from simply fine-tuning a strong TTS on the evaluation data. The concrete test I propose—a matched fine-tuned F5-TTS without the multi-modal conditions—would settle this directly. I do not see a separate internal inconsistency or mathematical error in the derivations; the concern is about experimental comparability and attribution. I therefore keep the reader's CONDITIONAL verdict unchanged, with the condition that the authors provide the matched baseline or release code and data to allow it to be run.","tokens_in":8930,"tokens_out":2732,"duration_ms":22418,"concrete_test":"Fine-tune the off-the-shelf F5-TTS model (or the same architecture) on the exact V2C-Animation, GRID, and proposed-dataset splits used in the paper, with the same reference-audio protocols (Dub 1.0, Dub 2.0, and Setting 3.0), the same training budget and duration-loss setup, but without the VLM conclusion, visual CLIP features, or ControlNet-style conditions. Evaluate this matched text-only F5-TTS baseline under the metrics of Tables 1-5. If this baseline matches or exceeds the proposed model's SPK-SIM, EMO-SIM, MCD, and LSE-D, then the paper's central claim about multi-modal conditions collapses; if the proposed model still clearly beats the matched baseline, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the proposed multi-modal framework—VLM-derived dubbing type and fine-grained attribute conditions plus ControlNet-style visual conditioning—outperforms SOTA methods. The most load-bearing unsupported assumption is the comparability of the F5-TTS baseline in Tables 1-3. The proposed model inherits the F5-TTS architecture, is pre-trained on Emilia-EN in Stage 2.1, and is then fine-tuned on V2C-Animation, GRID, and the proposed movie dubbing dataset in Stage 2.2. The F5-TTS row, however, appears to be the off-the-shelf pretrained model with no fine-tuning on any of the evaluation datasets. Because the proposed model has seen the target-domain data during training and the baseline has not, the observed improvements in SPK-SIM, EMO-SIM, and MCD could be explained entirely by domain adaptation rather than by the multi-modal understanding and conditioning that the paper motivates as its contribution. The paper provides no matched baseline, no error bars on objective metrics, and no ablation in which a text-only F5-TTS is fine-tuned on the same data with the same budget. The existing ablation in Table 6 shows that removing the video conclusion changes SPK-SIM by only 0.40 and EMO-SIM by 1.56, which further suggests the VLM conditions contribute little relative to the gap between the proposed model and the unmatched F5-TTS baseline. Without a matched fine-tuned F5-TTS comparison, the attribution of the gains to the multi-modal design is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-MovieDubber, a two-stage multi-modal framework for movie dubbing. In Stage 1, a vision-language model is trained (via instruction tuning) to analyze silent video and produce an 'understanding conclusion' that includes dubbing type (dialogue/narration/monologue) and fine-grained speaker attributes (gender, age, emotion). In Stage 2, a diffusion-transformer speech generator (architecture inherited from F5-TTS) is pre-trained on Emilia-EN and then tuned with ControlNet-style multi-modal conditioning using video features, the VLM conclusion, and the script. The authors also construct a 7.2-hour annotated movie dubbing dataset. Experiments on V2C-Animation, GRID, a zero-shot setting, and the proposed dataset report improvements over several dubbing baselines on metrics including LSE-D, SPK-SIM, EMO-SIM, and MCD, with the largest gains on the proposed dataset.","tokens_in":9282,"tokens_out":8372,"duration_ms":59315,"significance":"If the claimed improvements are robust and correctly attributed, the framework would be a useful contribution to movie dubbing, particularly the ability to dub without a reference speech and to condition on scene type and speaker attributes. The paper's strengths include evaluation on external benchmarks, a newly constructed annotated dataset, and a zero-shot experimental setting that is practically relevant. However, the central claim that the multi-modal VLM/ControlNet conditions are the source of the gains is not adequately supported: the F5-TTS baseline appears unmatched in training data, the ablation differences are small, and the 'superior performance' claim is contradicted by the GRID results. The experimental protocol is also ambiguous regarding which datasets are used for the fine-tuning stage.","major_comments":[{"comment":"The training and evaluation protocol is ambiguous, and the F5-TTS baseline is not matched in training data. Section 3.2 states that the speech-generation tuning stage (Stage 2.2) is implemented with only the proposed movie-dubbing dataset, yet Tables 1 and 2 report results on V2C-Animation and GRID. It is not stated whether the proposed model was fine-tuned on those benchmarks' training splits or evaluated in a zero-shot/cross-dataset manner, nor whether the F5-TTS row in Table 1 is the off-the-shelf Emilia-pretrained model or a model fine-tuned on dubbing data. If the proposed model received additional training on movie dubbing data (even from a different dataset) while F5-TTS did not, the gains over F5-TTS (e.g., WER 22.52 vs 24.41 in Table 1) could be due to domain adaptation rather than to the VLM-derived conditions and ControlNet. The paper must clarify the protocol and provide a matched comparison, such as fine-tuning F5-TTS on the same dubbing data and budget, or evaluating an ablated version of the proposed model without Stage 2.2, to support the attribution.","section":"Section 3.2 and Tables 1-2"},{"comment":"The claim of 'superior performance compared to state-of-the-art (SOTA) methods' is overstated and not supported by the GRID results. In Table 2, the proposed method is worse than F5-TTS on WER (14.71 vs 11.94 in Dub 1.0) and SPK-SIM (95.73 vs 96.51), and in Table 3 it is worse than F5-TTS on both NMOS (4.33 vs 4.43) and SMOS (3.14 vs 3.32). The abstract should be qualified to specify the metrics and datasets for which improvements hold, and the main text should acknowledge that the method does not dominate F5-TTS on GRID. The current wording implies global superiority, which is inaccurate.","section":"Abstract and Section 4.1.1"},{"comment":"The ablation study does not establish that the multi-modal conditions are the driver of the reported improvements. Removing the video conclusion changes SPK-SIM by only 0.40 (82.99 vs 82.59) and EMO-SIM by 1.56 (64.74 vs 63.18), and removing the video clip changes SPK-SIM by 0.36. No error bars or significance tests are provided, yet the text claims that removing video clip control 'causes all metrics to drop significantly' and that the conclusion control 'improves SPK-SIM and EMO-SIM.' Given the small effect sizes relative to the gaps over baselines in Tables 1 and 2 (e.g., SPK-SIM gains of 0.44 over F5-TTS on V2C-Animation), the attribution of the main benchmark improvements to the VLM conclusion and visual conditioning is not supported.","section":"Table 6"},{"comment":"The paper's novelty rests on fine-grained attribute understanding, but no experiment isolates the contributions of scene type (Cs), gender (Cg), age (Ca), and emotion (Ce). In Eq. (6), these are concatenated into a single condition Cv, and the ablation in Table 6 removes the entire 'conclusion' rather than individual attributes. Consequently, the specific value of each fine-grained attribute for dubbing quality is not demonstrated. The authors should provide an ablation that removes each attribute separately or, at minimum, report the model's accuracy in predicting these attributes and its effect on the final output.","section":"Section 2.3.2 and Table 6"}],"minor_comments":[{"comment":"There are several typos: 'Curent' in Figure 1, 'decreaded' in the V2C-Animation results paragraph, and 'SIP-SIM' for 'SPK-SIM' in the zero-shot results paragraph. The heading of Table 5 reads 'towards fined-grained movie dubbing test' and should be corrected to 'toward fine-grained movie dubbing test.'","section":"Figure 1 and Section 4.1.1"},{"comment":"Equation (9) mixes the symbols g_theta and v0 for the model output; the classifier-free guidance formula should use consistent notation, e.g., v_theta throughout, to avoid confusion about the velocity estimate being combined.","section":"Equation (9)"},{"comment":"References [4] and [11] are the same paper (Cong et al., 'Learning to dub movies via hierarchical prosody models'), and references [1] and [22] are also the same work (Chen et al., V2C). These duplicates should be consolidated.","section":"References"},{"comment":"The sentence 'slightly lower than the best fine-tuned F5-TTS model, Speaker2Dub and StyleDubber' is unclear; it appears to mean the proposed method's WER is slightly higher than the best values among F5-TTS, Speaker2Dub, and StyleDubber. This should be rephrased for clarity.","section":"Section 4.1.1, GRID paragraph"},{"comment":"The description of how the mLLM is trained in Stage 1 (Section 2.2) and how its conclusions are generated is underspecified: no details are given about the base mLLM, the instruction template, the amount of training data, or the annotation process for the proposed dataset. Adding these details would improve reproducibility.","section":"Section 2.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent system paper with a promising idea, but the experimental validation currently has a load-bearing confound: the F5-TTS baseline appears not to be fine-tuned on the dubbing data, while the proposed model is, making the attribution of gains to the multi-modal conditions uncertain. The ablation effect sizes are small, and the abstract overstates the results given the GRID numbers. With a matched baseline, clearer protocol, and stricter claims, the paper could become a solid contribution to ACM MM or a similar venue. In its current form, I cannot recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of 2505.16279. The useful part is the integration: a VLM extracts dubbing type (dialogue/narration/monologue) plus gender, age, emotion from video, and those become explicit conditions in an F5-TTS-style flow-matching generator with ControlNet-style multimodal conditioning. The 7.2-hour annotated dataset is a real artifact, and the zero-shot result on V2C using GRID reference voices (SPK-SIM 83.55, WER 15.49) is the strongest evidence the system generalizes. Against the other dubbing systems trained on the same benchmarks, the V2C-Animation numbers look competitive.\n\nThe soft spot is exactly what the stress-test note flags. The F5-TTS row in Tables 1 and 2 appears to be the off-the-shelf Emilia model, not fine-tuned on V2C-Animation or GRID. The proposed model is fine-tuned on those datasets plus the new one before evaluation. So the gains over F5-TTS—up to 8.80% SPK-SIM, 19.08% EMO-SIM—could come from domain adaptation alone. There is no matched baseline: a text-only F5-TTS fine-tuned on the same data with the same budget. Their ablation (Table 6) makes the worry concrete: removing the video conclusion changes SPK-SIM by 0.40 and EMO-SIM by 1.56, far smaller than the gap to the unmatched baseline. The attribution of the gains to the VLM/ControlNet design is therefore not established.\n\nOn GRID, Tables 2 and 3 show the proposed method is worse than F5-TTS on WER (14.71 vs 11.94 under Dub 1.0), SPK-SIM (95.73 vs 96.51), and NMOS (4.33 vs 4.43). The text acknowledges this but still claims superior performance, which is too generous.\n\nMinor issues: no error bars on objective metrics, no code or data release, and the proposed dataset test (Section 4.1, \"towards fined-grained\") is new territory for all systems, so it is more of a zero-shot stress test than a clean benchmark. Also, HPMDubbing's WER on V2C is 150%, which suggests some evaluation irregularity worth checking.\n\nOverall, this is a legitimate integration paper with a useful dataset. If the authors add a matched fine-tuned F5-TTS baseline, release code/data, and temper the SOTA wording, it would be a solid contribution. For an editor: send to peer review; the dataset and pipeline deserve scrutiny, but the referee should require the matched baseline before acceptance.","headline":"Useful VLM-conditioned dubbing integration with a real annotated dataset, but the SOTA claim rests on an unmatched F5-TTS baseline.","tokens_in":9808,"tokens_out":3308,"would_cite":false,"duration_ms":23368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Movie dubbing that reads scene type and speaker attributes from video outperforms prior state-of-the-art dubbing models on sync, timbre, and emotion.","keywords":["movie dubbing","multi-modal learning","vision-language model","speech generation","diffusion transformer","flow matching","audio-visual synchronization","dubbing dataset"],"falsifier":"Fine-tune F5-TTS on the same V2C-Animation and GRID training splits with the same reference-audio settings and duration loss, and compare it with MM-MovieDubber when the VLM conclusion is ablated; if the fine-tuned F5-TTS matches or exceeds the reported SPK-SIM, EMO-SIM, and MCD improvements, the central claim that the video-understanding conditions drive the gains is contradicted.","tokens_in":8748,"feed_emoji":"🎬","tokens_out":8384,"duration_ms":58687,"temperature":0.7,"pith_summary":"The paper aims to establish that movie dubbing improves when the dubbing system explicitly understands the scene and the speaker rather than relying only on a reference voice and low-level video features. The proposed framework, MM-MovieDubber, first uses a multi-modal vision-language model to turn a silent video clip into a structured conclusion: scene type (dialogue, narration, or monologue) and fine-grained speaker attributes (gender, age, emotion). A diffusion-transformer speech generator, built on the F5-TTS architecture, is then conditioned on this conclusion, the video frames, and the script, with a duration loss for synchronization and classifier-free guidance over all three conditions. The authors also contribute a 7.2-hour movie dubbing dataset annotated with these dubbing types and attributes. Across the V2C-Animation and GRID benchmarks, plus zero-shot and no-reference-reference settings, they report improvements over prior state-of-the-art dubbing models in lip-sync distance, speaker similarity, emotion similarity, and mel-cepstral distortion, up to 1.09%, 8.80%, 19.08%, and 18.74% respectively.","feed_headline":"Dubbing that reads the scene beats SOTA on voice and emotion","feed_subtitle":"A vision-language model labels dubbing type, gender, age, and emotion — and dubbing works even without a reference voice.","key_machinery":"The key mechanism is a two-stage generative pipeline. Stage 1 is a multi-modal large language model (mLLM) trained with instruction tuning to convert a silent video clip into a structured 'understanding conclusion' $C_v = \\{C_s, C_g, C_a, C_e\\}$ covering scene type, speaker gender, speaker age, and emotion. Stage 2 is a diffusion-transformer speech generator (the F5-TTS architecture with conditional flow matching) whose denoising is conditioned on CLIP-encoded video frames, the conclusion, and the script, using cross-attention to fuse the conclusion and visual features. A duration loss constrains speech-video length consistency, and inference uses classifier-free guidance over video, conclusion, and script conditions.","core_discovery":"The central claim is that conditioning speech generation on a vision-language-model-derived 'understanding conclusion' — composed of scene type, speaker gender, speaker age, and voice emotion — yields dubbing that is better synchronized, more similar to the target voice, and more emotionally expressive than state-of-the-art dubbing models, including HPMDubbing, StyleDubber, and Speaker2Dubber. The paper further claims this works without a reference voice: given a silent video and script alone, the model produces speech that is intelligible and emotionally appropriate. Supporting evidence includes objective metrics on V2C-Animation and GRID and subjective MOS, with gains such as SPK-SIM rising from 79.81% to 83.30% and EMO-SIM from 59.71% to 64.93% in the Dub 2.0 setting on V2C-Animation, and MCD falling from 9.11 to 8.80.","pith_inferences":["A controlled comparison that fine-tunes the same F5-TTS backbone on the identical dubbing data, with and without the VLM conclusion, would isolate how much of the reported gain comes from the understanding conditions versus from fine-tuning on in-domain audio-video pairs.","Because the 'understanding conclusion' is an interpretable intermediate representation, its accuracy against human labels for scene type, gender, age, and emotion could be measured directly; if speech improvements do not track correct labels, the conditioning mechanism would need a different explanation.","The same scene-type and attribute conditions could serve as a shared semantic control layer for dubbing into other languages or for generating character-specific voices, since the conditions are language-agnostic."],"forward_implications":["Dubbing without a reference voice: silent video plus script is enough to produce intelligible, emotionally appropriate speech, as shown in the proposed fine-grained test.","Explicit scene-type and speaker-attribute labels act as effective conditioning, reflected in improved EMO-SIM and SPK-SIM and reduced MCD on both V2C-Animation and GRID.","The new annotated dataset provides a benchmark for dubbing that distinguishes dialogue, narration, and monologue and fine-grained attributes, which existing benchmarks do not cover."],"supporting_citations":[{"why":"Supplies the F5-TTS diffusion-transformer architecture and flow-matching backbone the speech generator is built on.","marker":"[16]"},{"why":"A primary state-of-the-art movie dubbing baseline it compares against and the source of the SPK-SIM evaluation approach.","marker":"[12]"},{"why":"A primary baseline and provider of the evaluation settings (Dub 1.0, Dub 2.0, and zero-shot reference) used in the comparison.","marker":"[13]"},{"why":"Provides the Emilia-EN speech corpus used to pre-train the foundational TTS model.","marker":"[21]"},{"why":"Supplies the V2C-Animation benchmark and the dub settings used for training and evaluation.","marker":"[22]"},{"why":"Supplies the GRID dubbing benchmark, the basis for the lip-sync experiments.","marker":"[23]"},{"why":"Provides the conditional flow-matching objective that trains the speech generation model.","marker":"[17]"},{"why":"Supplies the CLIP visual encoder used to derive video-frame features for conditioning.","marker":"[19]"},{"why":"Provides the classifier-free guidance formulation used in the modified velocity estimate during inference.","marker":"[20]"}],"fun_headline_variants":["VLM-guided dubbing beats SOTA on emotion and voice sync","Dubbing without reference voice: multi-modal model wins on emotion","Scene-reading dubbing: better voice, emotion, and sync than SOTA","Multi-modal movie dubbing: vision model boosts emotion by 19%","Dubbing that understands scenes: no reference voice, higher scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's comparison assumes the F5-TTS baseline is the off-the-shelf Emilia-trained model, while the proposed model is additionally trained on the dubbing datasets; if most of the reported gains come from that fine-tuning alone and not from the VLM-derived dubbing-type and attribute conditions, the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["VLM-guided dubbing beats SOTA on emotion and voice sync","Dubbing without reference voice: multi-modal model wins on emotion","Scene-reading dubbing: better voice, emotion, and sync than SOTA","Multi-modal movie dubbing: vision model boosts emotion by 19%","Dubbing that understands scenes: no reference voice, higher scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":4074,"prompt_tokens":963,"completion_tokens":3111,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3019}},"tokens_in":579,"tokens_out":3111,"duration_ms":20484,"temperature":1.0,"reasoning_tokens":3019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:03:46.931007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune F5-TTS on the same V2C-Animation and GRID training splits with the same reference-audio settings and duration loss, and compare it with MM-MovieDubber when the VLM conclusion is ablated; if the fine-tuned F5-TTS matches or exceeds the reported SPK-SIM, EMO-SIM, and MCD improvements, the central claim that the video-understanding conditions drive the gains is contradicted.","supporting_citations":[{"cited_title":"Learning to dub movies via hierarchical prosody models,","cited_arxiv_id":null,"evidence_quote":"Supplies the F5-TTS diffusion-transformer architecture and flow-matching backbone the speech generator is built on."},{"cited_title":"Mcdubber: Multimodal context-aware expressive video dubbing,","cited_arxiv_id":null,"evidence_quote":"A primary state-of-the-art movie dubbing baseline it compares against and the source of the SPK-SIM evaluation approach."},{"cited_title":"Audiopedia: Audio qa with knowledge,","cited_arxiv_id":null,"evidence_quote":"A primary baseline and provider of the evaluation settings (Dub 1.0, Dub 2.0, and zero-shot reference) used in the comparison."},{"cited_title":"Flow matching for generative modeling,","cited_arxiv_id":null,"evidence_quote":"Supplies the V2C-Animation benchmark and the dub settings used for training and evaluation."},{"cited_title":"Visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP visual encoder used to derive video-frame features for conditioning."}],"review_version":1}