{"id":"7ac6e9f7-c820-47b2-ae15-607141317c50","arxiv_id":"2507.10109","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single multimodal language model can generate intelligible speech and synchronized background audio jointly from a silent video, transcript, and reference voice, outperforming separately concatenated speech and audio models on a new benchmark.","lead":"DualDub is a system that takes a silent video, a transcript, and a voice sample, and produces both a speech track and matching background audio at the same time. It is the first unified model for what the authors call video-to-soundtrack generation, and it comes with a new benchmark for judging how well the speech and audio fit together.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DualScore's harmony margin may reflect training/eval distribution overlap from the same separation pipeline; per-item validation against human ASCH is missing.","rationale":"The paper's strongest claim is that DualDub improves audio-speech harmony over concatenated independent V2A and V2S models. For that claim to hold, the benchmark's ground-truth separation must be clean and the new DualScore must track human judgments of harmony. The reader identified exactly this evaluation-pipeline dependency. My pass sharpens it: the CASP metric is trained on pairs generated by the same separator used to construct the test set, and its only reported validation is retrieval on that same distribution plus a coarse four-system subjective comparison. The high ground-truth WER and the authors' own limitation note independently corroborate that the separation pipeline is imperfect. A per-clip correlation between DualScore and human ASCH would settle the question directly; without it, the 0.59-versus-0.19-0.30 margin is not yet established as perceptually meaningful. The subjective ASCH results (3.44 vs. 1.44-1.69) are encouraging and prevent this from being a fatal flaw, but they are based on only 16 clips and do not by themselves validate the metric. The architecture, curriculum, and released benchmark are real contributions, so a conditional verdict remains appropriate rather than rejection.","tokens_in":22559,"tokens_out":7616,"duration_ms":99977,"concrete_test":"Take the released DualBench test set and system outputs, stratify approximately 60 clips across the observed DualScore range, and collect per-clip ASCH ratings from at least 20 listeners. Compute the clip-level Spearman correlation between DualScore and mean ASCH. If rho is below 0.3 or not significantly positive, DualScore does not reliably measure perceived audio-speech harmony, and the DualDub-versus-baseline margin in Table 5 cannot be interpreted as a harmony improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative support for DualDub's harmony advantage is the DualScore gap in Table 5 (0.59 vs. 0.19-0.30), but that gap is only interpretable if DualScore measures perceived audio-speech harmony. Two load-bearing links are unestablished. First, the 1,500-hour private CASP training set and the V2ST test set are produced by the same Mel-RoFormer separation and -40 dB filtering pipeline (SS4.2, SS5.1). If that separator leaves correlated artifacts (speech bleed into the background track, common spectral coloration) in both training and test pairs, the CASP audio encoder can learn to reward those artifacts rather than genuine harmony. The paper's own Table 3 shows ground-truth WER of 35.62% versus 12.74% for DualDub, indicating the separated speech references are not clean, and Section 8 concedes Mel-RoFormer 'sometimes separates empty audio or speech segments.' Second, the only validation of DualScore is Top-k retrieval on the same V2ST test set (SS4.3) plus a system-level comparison with ASCH using just 4 systems and 16 clips (Table 5). No per-clip correlation between DualScore and human ASCH is reported, so the system-level ordering could be driven by a few extreme items or by the same separation artifacts. If DualScore is artifact-driven, the objective harmony claim collapses even though the subjective trend is suggestive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, video-to-soundtrack (V2ST) generation, and presents DualDub, a multimodal language model that jointly generates background audio and speech from video, transcript, and a reference speech clip. The model consists of a multimodal encoder, a cross-modal aligner with causal and non-causal attention, dual decoding heads, and a curriculum learning strategy. The authors also introduce DualBench, a benchmark with a curated test set built from V2C-Animation via Mel-RoFormer separation and energy filtering, plus metrics including the newly proposed DualScore for audio-speech harmony. Experiments compare DualDub against concatenated V2A and V2S baselines, reporting higher objective harmony scores and higher subjective audio-speech correspondence and harmony (ASCH), along with V2A/V2S quality comparisons and ablations.","tokens_in":22754,"tokens_out":3082,"duration_ms":36046,"significance":"If the central empirical claim is accepted, the paper makes a useful contribution by defining a practical and underexplored task, proposing a coherent architecture for joint speech and background audio generation, and releasing a benchmark with code. The consistent subjective trend (Table 5, ASCH 3.44 vs 1.44-1.69) is a genuine strength, as is the open benchmark and the detailed curriculum learning design. However, the main quantitative evidence for the 'harmony' advantage is DualScore, a metric introduced and trained in this paper on private data processed by the same separation pipeline used to build the benchmark test set; no human correlation validation is provided. This makes the objective pillar of the central claim currently unverified, although the issue is addressable with additional evaluation.","major_comments":[{"comment":"The load-bearing objective evidence for the harmony advantage is DualScore, but its validity as a measure of perceived audio-speech harmony is not established. The CASP model is trained on 1,500 hours of private audio-speech pairs created with the same Mel-RoFormer separation and -40 dB filtering pipeline used to build the V2ST test set (§4.2, §5.1), and its only reported validation is retrieval accuracy on that same V2ST test set. No per-clip correlation between DualScore and human ASCH ratings is reported, so the large DualScore gap (0.59 vs 0.19-0.30) could reflect separator-specific artifacts, speech bleed, or other correlated features rather than harmony that humans perceive. I ask the authors to validate DualScore against human harmony judgments on a held-out set (including per-item correlation), to release the CASP training data or otherwise remove the training/evaluation overlap, and to re-report the system comparison using an independently separated or manually verified test set.","section":"§4.3, §5.1, Table 5"},{"comment":"The ground-truth word error rate on the DualBench test set is 35.62%, while DualDub's generated speech reaches 12.74%; this indicates that the separated speech references in the test set are heavily corrupted or mislabeled, contradicting the assumption of clean, correctly paired ground-truth speech. Section 8 itself concedes that Mel-RoFormer 'sometimes separates empty audio or speech segments.' This does not necessarily invalidate the comparison across systems, but it undermines the reliability of ground-truth-anchored metrics and makes the test set unsuitable as a benchmark reference without further quality control. The authors should quantify the proportion of corrupted clips, provide a manual verification subset, and report the main results on that subset.","section":"Table 3, §8"},{"comment":"Objective metrics are reported as point estimates without error bars, confidence intervals, or significance tests. For instance, the DualScore differences in Table 5 (0.59 vs 0.19-0.30) and the FD/FAD/KLD differences in Tables 2 and 4 are presented without any measure of variance, so the reader cannot assess whether the differences are within noise. The subjective ASCH comparison is based on only 16 clips, even though 1,319 test clips are available. I request bootstrap confidence intervals or significance tests on all objective metrics and a subjective evaluation on a larger, diverse sample, or at least a justification for the small subjective sample size.","section":"Tables 2-5"}],"minor_comments":[{"comment":"The label 'Casual Non-Casual' should be 'Causal Non-Causal'.","section":"Figure 2"},{"comment":"The phrase 'we access audio and speech quality' should be 'we assess audio and speech quality'.","section":"§4.3"},{"comment":"The reference list appears twice in the manuscript, duplicating references [1] through [58] verbatim; this should be corrected.","section":"References"},{"comment":"The column header 'SSIM↑' under 'Audio-Video alignment' appears to be a typo for 'SIM↑' (speaker similarity), since the value 0.84 matches the SIM column in Table 3 and SSIM is not defined in the text.","section":"Table 4"},{"comment":"Equation (6) contains a stray period and line break after the differential equation; the equation and surrounding punctuation should be cleaned up.","section":"Eq. (6)"},{"comment":"The sentence 'Notably, following the data process of DualBench, we separate all audio-speech streams to avoid speech leaks into the background audio' is grammatically awkward and should be revised for clarity.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is timely and the architecture is reasonable, but the evaluation pipeline has a non-trivial training/evaluation overlap for the headline metric. The paper would be substantially strengthened by releasing the CASP training data or its separation settings, validating DualScore against human ratings per item, and reporting uncertainty on all objective metrics. The duplicated reference list and the extremely high ground-truth WER also need attention before this is suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely new: it defines video-to-soundtrack generation (speech plus background audio from a silent video), builds a unified model for it, and releases a benchmark. The architecture is reasonable — a multimodal LM with dual heads, a causal/non-causal aligner, and a three-stage curriculum that sensibly reuses V2A and TTS data. The ablations are informative, particularly the aligner's effect on AV-Align and DualScore. The authors are also honest about known weaknesses: Section 8 concedes that WavTokenizer loses acoustic detail and that Mel-RoFormer separation produces empty segments. That candor counts for something.\n\nThe soft spots are real and mostly evaluative. DualScore, the objective 'harmony' metric that carries the main claim, is trained on 1,500 hours of private audio-speech pairs produced by the same Mel-RoFormer separation and -40 dB filtering used to build the test set. The paper does not show that the separator leaves no correlated artifacts (speech bleed, spectral coloration). With ground-truth WER at 35.62% versus 12.74% for the model, the references themselves are clearly not clean, so the metric could be rewarding separation artifacts rather than perceived harmony. The only external check is a system-level comparison with human ASCH scores on 4 systems and 16 clips; there is no per-clip correlation between DualScore and human ratings. That makes the 0.59 versus 0.19–0.30 gap look fragile, even though the subjective trend points in the same direction.\n\nOther issues: objective metrics in Tables 2–4 lack error bars and significance tests, and on VGGSound DualDub trails MMAudio on audio quality, so 'state-of-the-art' is only true on the authors' own benchmark. None of this sinks the paper, but it does mean the central superiority claim is not yet established.\n\nVerdict: worth engaging. The task and model are useful contributions for the multimedia and speech-generation community, and the benchmark can become a shared resource. A serious referee should ask for External validation of DualScore (per-item correlation with human judgments on a held-out set, or at least a cross-dataset sanity check), error bars on the objective numbers, and a cleaner test set. With those, the paper would be solid. I would accept it for peer review.","headline":"A credible new task and a well-built unified model, but the harmony metric that carries the headline claim needs external validation before the superiority result lands.","tokens_in":23385,"tokens_out":1599,"would_cite":true,"duration_ms":20436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes video-to-soundtrack generation, in which a single multimodal language model produces both intelligible speech and background audio from a video, and reports that this joint approach yields markedly more harmonious and…","keywords":["video-to-soundtrack generation","joint speech and audio synthesis","cross-modal alignment","curriculum learning","multimodal language model","audio-speech harmony","dubbing","video-to-audio generation"],"falsifier":"An audit of the test set would settle it: if listeners can hear intelligible speech leaking into the separated 'background' tracks, or if a blind forced-choice test finds the stitched baselines as harmonious as DualDub as often as not, the reported harmony advantage is not established.","tokens_in":22266,"feed_emoji":"🎬","tokens_out":7135,"duration_ms":77961,"temperature":0.7,"pith_summary":"Existing video-to-audio systems can synthesize background effects but treat speech as just another sound, producing unintelligible vocal noise. This paper proposes video-to-soundtrack generation as a distinct task: producing intelligible, speaker-controlled speech and background audio together from a video, a transcript, and a reference voice. It introduces DualDub, a multimodal language model with a cross-modal aligner that predicts audio and speech tokens in a single autoregressive pass, trained with a curriculum schedule that sidesteps the scarcity of fully paired data. On the new DualBench benchmark, DualDub reports substantially higher audio-speech harmony than concatenating separate state-of-the-art video-to-audio and video-to-speech systems. The claim matters because it suggests single-pass joint generation, rather than post-hoc mixing, is the right architecture for automated dubbing and narration.","feed_headline":"One model generates synchronized speech and background audio from video","feed_subtitle":"Joint generation scores 0.59 on a new harmony metric versus 0.19-0.30 for stitched baselines.","key_machinery":"The load-bearing components are three. The cross-modal aligner combines two causal cross-attention heads (audio attends to past speech, speech attends to past audio) with two non-causal heads (audio and speech each attend to the full video), so rhythm and timing cohere without leaking future information during generation. The curriculum schedule trains the model first on video-to-audio, then adds text-to-speech, then fine-tunes on a small amount of fully paired video-text-speech-audio data, which is what makes joint training feasible given data scarcity. The generation head is a multimodal language model with two output heads predicting audio and speech tokens simultaneously, decoded through a flow-matching network rather than the codec's native decoder to recover waveform quality. On the evaluation side, the paper introduces CASP, a contrastive audio-speech pretraining model whose cosine similarity between the audio and speech embeddings defines the DualScore harmony metric.","core_discovery":"The central claim is that generating background audio and speech in one autoregressive pass, conditioned on the same video, transcript, and speaker reference, yields soundtracks that are more synchronized and more acoustically harmonious than anything obtainable by running a separate video-to-audio model and a separate video-to-speech model and mixing their outputs. DualDub encodes video with a frozen visual encoder, tokenizes both tracks with a unified codec, lets causal cross-attention pass history between the audio and speech streams while non-causal attention aligns both with the video, and decodes tokens to waveforms through a flow-matching network into a pre-trained audio VAE latent space. The paper reports DualScore 0.59 versus 0.19-0.30 for concatenated baselines and subjective audio-speech correspondence and harmony scores of 3.44 versus 1.44-1.69, while remaining competitive or superior on individual audio and speech quality metrics. DualBench supplies 1,319 filtered test clips and a contrastive audio-speech pretraining metric, DualScore, for measuring harmony between the two generated tracks.","pith_inferences":["An extension the paper leaves implicit is that if DualScore tracks perceived harmony, it could be reused as a training reward for joint audio-speech models rather than only a test-time metric, since it is computed from frozen embeddings.","The paper's separation-and-filter pipeline suggests the quality ceiling of V2ST evaluation depends on source separation; stronger separation than the current model might enlarge the usable training set and change the measured harmony gap.","The stated bottleneck of discrete 40-token-per-second codec representation points to a concrete next experiment: replacing discrete tokens with continuous representations should improve speech naturalness and audio fidelity under the same curriculum.","A testable extension is to vary the speaker reference and transcript while fixing the video, checking whether the generated background audio adapts to the changed speech content rather than staying tied only to the visual scene."],"forward_implications":["For automated dubbing, the result implies a single model can replace separate audio-effects and speech models, removing the timing and level conflicts that appear when two independently generated tracks are mixed.","Because the curriculum learns from large existing video-audio and text-speech corpora plus a small paired set, other multimodal generation tasks facing the same data scarcity could adopt the same staged recipe.","DualBench gives later work a common yardstick, so quality, video synchronization, and audio-speech harmony can be measured on the same 1,319 clips.","Joint conditioning on video means speech prosody and background audio are shaped by the same visual events, enabling scene-driven narration to be produced end-to-end."],"supporting_citations":[{"why":"Supplies the V2C-Animation corpus whose separated test clips form the DualBench V2ST test set.","marker":"[3]"},{"why":"Separates speech from background audio; the test set and all training pairs depend on this separation.","marker":"[45]"},{"why":"Provides the unified codec that turns both background audio and speech into language-model tokens.","marker":"[24]"},{"why":"Provides the autoregressive language-model backbone that DualDub extends with dual output heads.","marker":"[42]"},{"why":"Baseline video-to-audio model used in the concatenated soundtrack comparisons.","marker":"[7]"},{"why":"Baseline video-to-audio model with temporal control used in the concatenated soundtrack comparisons.","marker":"[54]"},{"why":"Baseline video-to-speech model used in the concatenated soundtrack comparisons.","marker":"[55]"},{"why":"Baseline style-controlled dubbing model used in the concatenated soundtrack comparisons.","marker":"[11]"},{"why":"Provides the pre-trained audio VAE whose latent space the flow-matching decoder targets.","marker":"[16]"},{"why":"Initializes the audio branch of the CASP encoder used to compute the DualScore harmony metric.","marker":"[4]"}],"fun_headline_variants":["One model, one pass: speech and bg audio from video","Joint speech and background audio generation from video","DualDub: speech and bg audio in one unified model","Video to soundtrack: joint speech and bg audio beats stitching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's load-bearing premise is that the 1,319 test clips produced by automatic separation and a -40 dB energy cutoff contain clean, correctly paired speech and background audio, and that the learned DualScore matches what humans mean by harmony.","fun_headline_variants_meta":{"raw":{"variants":["One model, one pass: speech and bg audio from video","Joint speech and background audio generation from video","DualDub: speech and bg audio in one unified model","Video to soundtrack: joint speech and bg audio beats stitching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000893,"raw_usage":{"total_tokens":3857,"prompt_tokens":957,"completion_tokens":2900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2834}},"tokens_in":573,"tokens_out":2900,"duration_ms":22127,"temperature":1.0,"reasoning_tokens":2834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:40:26.611830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An audit of the test set would settle it: if listeners can hear intelligible speech leaking into the separated 'background' tracks, or if a blind forced-choice test finds the stitched baselines as harmonious as DualDub as often as not, the reported harmony advantage is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the V2C-Animation corpus whose separated test clips form the DualBench V2ST test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the unified codec that turns both background audio and speech into language-model tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the autoregressive language-model backbone that DualDub extends with dual output heads."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline style-controlled dubbing model used in the concatenated soundtrack comparisons."},{"cited_title":"In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained audio VAE whose latent space the flow-matching decoder targets."}],"review_version":1}