{"id":"16be39e4-b0ee-4f1f-bdaa-9ce49afbe51c","arxiv_id":"2508.00733","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single multimodal diffusion transformer generates video-synchronized general audio, speech, and song from flexible combinations of video, text, and lyrics inputs.","lead":"AudioGen-Omni is a single diffusion model that generates general sound effects, speech, and singing from a video, text, lyrics, or any combination. It claims top results on audio and speech benchmarks while running about four times faster than real time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states whether the LRS3/LRS2 speech evaluations feed the ground-truth transcription to the model; if they do, the reported WER 17.56/17.75 reflects text-to-speech, not video-to-speech, and the comparison with DiffV2S and Face2Voices is invalid.","rationale":"The central claim that AudioGen-Omni is the first unified video-synchronized audio/speech/song generator with state-of-the-art results rests heavily on the LRS3/LRS2 speech numbers, because those are the only quantitative results that clearly beat prior video-to-speech systems by a large margin. The paper's methodology never fixes the conditioning set for Table 2: Section 3.3 says inference can use arbitrary subsets of conditioning inputs, Section 3.2 introduces dense frame-level representations from grapheme/phoneme sequences, and Section 4.2.2 only describes how WER is computed, not what was fed to the model. If the ground-truth transcription was among the inputs, the 17.56 WER would no longer be evidence for visual speech inference, and the comparison with DiffV2S and Face2Voices would be invalid. This is exactly the weakest assumption identified by the reader, and a disclosure or a visual-only rerun would settle it. I note secondarily that the paper also contains no song evaluation, which weakens the 'song' leg of the unified claim, but the speech-conditioning protocol is the single most load-bearing issue because it can invalidate the strongest quantitative result. The current CONDITIONAL verdict already captures this uncertainty, so no change to the verdict is needed.","tokens_in":12865,"tokens_out":5687,"duration_ms":73582,"concrete_test":"Require the authors to rerun the LRS3/LRS2 speech evaluation with all text/transcription inputs masked so the model receives only the silent video (and any non-text global features), reporting WER, UTMOS, DNSMOS, and SECS in the same format as Table 2 and Table 3. If the WER degrades substantially from 17.56/17.75, the Table 2 result was conditioned on ground-truth text and the video-to-speech SOTA claim fails. An even simpler first check: inspect the released inference configuration or demo code to determine whether the LT-encoder inputs are empty during evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.2 reports WER 17.56/17.75 on LRS3/LRS2 and compares the result with video-to-speech baselines. Section 3.3 explicitly permits arbitrary subsets of conditioning inputs at inference, and Section 3.2 introduces a lyrics-transcription encoder that maps grapheme or phoneme sequences to dense frame-level representations. The evaluation paragraph in Section 4.2.2 specifies only how WER is computed; it never states whether the model received the ground-truth transcription, lyrics, or any other text input during the so-called video-to-speech evaluation. If the ground-truth transcript was supplied, the low WER is expected for a text-conditioned speech generator, and the comparison against DiffV2S and Face2Voices—which infer speech content from video alone—would not support the central claims of visual speech inference, flexible multimodal conditioning, or state-of-the-art video-to-speech performance. This is a missing-protocol concern, not an accusation: the paper must disclose the exact conditioning configuration for Table 2 and ideally ablate transcript-present vs. transcript-masked evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AudioGen-Omni, a 1.55B-parameter multimodal diffusion transformer for unified generation of general audio, speech, and song from flexible combinations of video, text, lyrics, and other conditions. The method combines a lyrics-transcription encoder, a joint attention mechanism, and a phase-aligned anisotropic positional infusion (PAAPI) scheme, and is trained jointly on large video-text-audio corpora with input masking and conditional flow matching. Experiments cover video-to-audio quality on VGGSound, video-to-speech quality on LRS3/LRS2, and speaker similarity on LRS3, with a reported inference time of 1.91 seconds for 8 seconds of audio. The paper claims state-of-the-art results on Text-to-Audio/Speech/Song tasks and the first unified framework for these generation types.","tokens_in":13124,"tokens_out":6029,"duration_ms":78273,"significance":"If the claims are substantiated, AudioGen-Omni would be a notable step toward a single architecture for general audio, speech, and singing generation with flexible multimodal conditioning, and the reported inference speed is practically attractive. The paper is also commendable for integrating a large-scale data pipeline, standard objective metrics, and a wide comparison set. However, the evidence as submitted is not yet sufficient for the central claims: the video-to-speech evaluation protocol is underspecified in a way that could invalidate the headline WER numbers, song generation is never quantitatively evaluated, the temporal-alignment metric shares the same pretrained network used for conditioning, and the claimed text-to-audio state of the art is not benchmarked. The architecture itself is reasonable and the results are promising, but the missing protocol and evaluation gaps are load-bearing.","major_comments":[{"comment":"The LRS3/LRS2 speech results are presented as video-to-speech performance, but the evaluation section never states which conditioning inputs were provided at inference. Section 3.3 explicitly permits arbitrary subsets of conditioning inputs, and Section 3.2 introduces a lyrics-transcription encoder that consumes grapheme/phoneme sequences. If the ground-truth transcription or lyrics were fed to the model, the reported WER of 17.56/17.75 would reflect text-conditioned speech generation rather than visual speech inference, and the comparison against DiffV2S and Face2Voices, which must infer speech content from video alone, would be invalid. The exact conditioning configuration for Table 2 must be disclosed, and ideally an ablation with and without transcript conditioning should be provided.","section":"§4.2.2, Table 2"},{"comment":"The abstract and introduction claim state-of-the-art results on Text-to-Audio/Speech/Song tasks and the first unified framework that generates song, but Section 4 contains no quantitative song-generation evaluation: there is no song benchmark table, no singing-quality metric, no lyric-intelligibility metric, and no comparison with song-generation baselines. The only song evidence is the demo link. Without a defined song evaluation protocol and quantitative results, the song-generation contribution and the corresponding SOTA claim cannot be assessed. A dedicated song evaluation with appropriate metrics and baselines is needed.","section":"§4.2, Abstract"},{"comment":"The temporal-alignment evaluation is partially circular. Section 4.1 uses the Synchformer DeSync score as the primary audio-visual synchronization metric, while Section 3.4.1 states that Synchformer's 24 FPS visual features are extracted and integrated into the model's global conditioning vector. The model is therefore trained and conditioned to align with Synchformer representations and then evaluated by the same pretrained network. This is especially consequential because the DeSync gap between AudioGen-Omni (0.450) and MMAudio (0.442) in Table 1 is very small. An independent temporal-alignment metric or human evaluation of synchronization is required to support the alignment claims.","section":"§4.1 vs §3.4.1"},{"comment":"The claim of state-of-the-art results on text-to-audio tasks is not supported by the presented evidence. Table 1 reports only VGGSound video-to-audio results; no text-to-audio benchmark such as AudioCaps or Clotho is included. Moreover, even on VGGSound, AudioGen-Omni is not uniformly better than MMAudio-L-44.1kHz: it improves FDPaSST and IS but is worse on FDPANNs (6.292 vs 4.72), KL divergence (1.556 vs 1.40), IB-score (29.261 vs 33.22), and DeSync (0.450 vs 0.442). The SOTA statement should be restricted to the settings actually measured, and the reported differences should be accompanied by significance or variance information.","section":"Abstract, Table 1"}],"minor_comments":[{"comment":"Several related-work citations appear to be mismatched: SongCreator is cited as Lei et al. (2020), which is an infectious-disease paper; MelodyLM is cited as Li et al. (2023), which points to an NTIRE super-resolution challenge paper; Freestyle is cited as Ning et al. (2006), which is an aluminum-alloys paper; and Yue is cited as Yuan et al. (2017), an energy-demand forecasting paper. These references need to be corrected.","section":"§2, References"},{"comment":"The sentence \"Our model outperforms existing VTS systems\" appears to contain a typo; it should probably read V2S systems.","section":"§4.2.2"},{"comment":"The text encoder is described as T5-Base in Section 3.2 but as Flan-T5 in Section 3.4.1 and Figure 1. Please clarify which encoder is actually used.","section":"§3.4.1 vs §3.2"},{"comment":"The DeSync score is described as estimating temporal misalignment in seconds, but Synchformer is a classifier over offset classes; the paper should specify how the offset class distribution is converted into a seconds-valued score and whether this conversion is the same as in prior work.","section":"§4.1"},{"comment":"The observation that the model surpasses ground-truth audio in UTMOS and DNSMOS is attributed to clean-speech generation, but these are non-intrusive quality predictors and may favor clean synthetic audio over noisy real recordings. The claim should be framed more cautiously, and a human listening test would strengthen it.","section":"§4.2.2"},{"comment":"No code, model weights, or demo evaluation details are provided beyond the demo link. For a system paper of this scope, releasing the inference code or at least a detailed evaluation configuration would greatly improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The central architectural idea is plausible and the inference speed is attractive, but the manuscript's headline claims currently outrun its evidence. The most urgent request is the exact inference-protocol disclosure for Table 2: if ground-truth transcripts were used, the reported WER numbers and the comparison with video-only baselines would need to be reframed substantially. The complete absence of quantitative song evaluation is also surprising given that song generation appears in the title and abstract. The DeSync circularity should be addressed with an external metric. I would be willing to review a revised version with these gaps filled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe reader's conditional verdict is about right. The genuinely new piece is the combination: a single 1.5B multimodal diffusion transformer that does general audio, speech, and song from video, text, or lyrics, with a duration-free lyrics-transcription encoder and selective RoPE (PAAPI) for temporal alignment. That combination is not in the cited prior work. The engineering is credible—training data and budget are specified, 25-step inference runs at 1.91s for 8s of audio, and the VGGSound numbers are competitive with MMAudio on DeSync while the LRS3 WER is much lower than the video-to-speech baselines. If those speech numbers hold up, that is a real step.\n\nTwo things keep me from trusting the headline claims as written. First, the speech evaluation protocol is not disclosed. Section 4.2.2 reports WER 17.56/17.75 on LRS3/LRS2 and compares against DiffV2S and Face2Voices, both of which infer speech content from video alone. The paper never states whether the model received the ground-truth transcription at inference. Given that Section 3.3 explicitly allows arbitrary conditioning subsets and the model ships a lyrics-transcription encoder, this is critical. If the transcript was provided, the comparison is text-to-speech, not video-to-speech, and the WER drop is expected. The stress-test note is right: the authors must disclose the exact conditioning inputs for Table 2 and ideally ablate transcript-present vs. transcript-masked evaluation.\n\nSecond, the song claim has no quantitative support. The abstract and conclusion claim SOTA on Speech/Song, but there is no song evaluation table anywhere—only a 1,000-hour lyrics dataset and presumably demo clips. Similarly, the claimed Text-to-Audio SOTA is supported only by a video-to-audio VGGSound table, with no text-only audio benchmark. The claims exceed the evidence.\n\nThere is also a minor circularity worth noting: the DeSync temporal metric is computed by Synchformer, and the model uses Synchformer's 24 FPS features as conditioning. That is not airtight; an external sync metric would strengthen the alignment claim. The UTMOS/DNSMOS above ground-truth is reasonably attributed to clean-speech artifacts, but it deserves a caption note.\n\nMy take: the architecture is real, and the V2S result is potentially important. But the missing protocol details and unsupported song claims mean this needs major revision before the SOTA statements can be taken at face value. Send it to peer review; a serious referee should insist on the disclosure and the song numbers. Not a desk reject.","headline":"Real unified model with strong V2S numbers, but undisclosed speech conditioning and unsupported song claims make the SOTA headline premature.","tokens_in":13656,"tokens_out":4350,"would_cite":false,"duration_ms":51383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single diffusion transformer claims to generate video-synchronized audio, speech, and song, setting new state-of-the-art results on each task.","keywords":["multimodal diffusion transformer","video-to-audio generation","video-to-speech generation","song generation","audio-visual synchronization","flow matching","lyrics-transcription encoder","positional embedding"],"falsifier":"Run the model on the LRS3 test set twice: once with only the silent video as conditioning and once with the ground-truth transcript added. If the transcript-free WER is much higher than 17.56 while the transcript-conditioned WER matches it, the claim of visual speech inference collapses; if both are near 17.56, the video-only premise holds.","tokens_in":12673,"feed_emoji":"🎵","tokens_out":8437,"duration_ms":84079,"temperature":0.7,"pith_summary":"AudioGen-Omni is a single multimodal diffusion transformer trained to produce general audio, speech, and song from text, video, or both, synchronised to the visual track. The paper claims it is the first unified framework to cover all three audio types with flexible conditioning, and that one checkpoint reaches state-of-the-art quality on text-to-audio, video-to-audio, video-to-speech, and song generation. The main technical bets are a duration-free lyrics-transcription encoder and phase-aligned anisotropic positional infusion (PAAPI), which applies rotary position embeddings selectively to time-structured modalities. If correct, a single model replaces separate audio, speech, and song systems, and video-to-speech intelligibility improves substantially at an inference cost of about 1.9 seconds for 8 seconds of audio.","feed_headline":"One model generates video-synced audio, speech, and song","feed_subtitle":"Its video-to-speech WER drops to 17.56 from the prior best of 30.37 on LRS3.","key_machinery":"The load-bearing object is the multimodal diffusion transformer block with joint attention: query, key, and value vectors from text, audio, and video are concatenated, attended over together, and then split back by modality, so every stream can read every other stream in one pass. Three mechanisms carry the argument. The lyrics-transcription encoder maps graphemes and phonemes to dense frame-level tokens using VoiceBPE tokenization and ConvNeXt refinement, with no phoneme-duration supervision. PAAPI (phase-aligned anisotropic positional infusion) adds rotary position embeddings only to video, audio, and time-aligned text streams, leaving atemporal text with plain positional embeddings, so time-indexed positions line up across streams during joint attention. Conditional flow matching trains a velocity field with variable-length masking, so the model learns to work with any subset of conditioning inputs at inference.","core_discovery":"The central claim is that a single flow-matching diffusion transformer with joint attention over video, text, and audio latents can be trained end-to-end on a mixture of video-text-audio, audio-text, and song-lyrics corpora, and that this joint training yields state-of-the-art results across generation tasks. Concretely, on VGGSound the model reports Fréchet distances of 58.766 (PaSST) and 6.292 (PANNs) with a DeSync of 0.450 seconds; on LRS3 and LRS2 it reports word error rates of 17.56 and 17.75, below the previous best video-to-speech systems; and the same checkpoint handles song generation through a lyrics-transcription encoder. The authors attribute these results to unfreezing all modalities, masking absent inputs during training, and giving temporally structured streams their own rotational positional phase.","pith_inferences":["If the reported video-to-speech numbers were produced with ground-truth transcripts available as conditioning, the WER gain would reflect text-to-speech rather than visual speech inference; the paper does not explicitly rule this out, so a transcript-free rerun is the decisive check.","The paper presents no quantitative song-generation benchmark, so the 'song' claim currently rests on the lyric dataset and encoder design rather than a measured comparison against song-synthesis baselines.","The same joint-attention architecture could be turned toward video generation, as the conclusion hints, by replacing the audio latent target with a visual one; the PAAPI alignment mechanism would then need to synchronise text and audio to the generated frames."],"forward_implications":["A single 1.55-billion-parameter checkpoint covers text-to-audio, video-to-audio, video-to-speech, and song generation, so downstream video production can drop task-specific pipelines.","Video-to-speech intelligibility on LRS3 improves from a previous best WER of 30.37 to 17.56, a large step toward lip-synchronised dubbing.","Generating 8 seconds of audio takes about 1.91 seconds at 25 sampling steps, making near-real-time use plausible.","Because absent inputs are masked during training, the same model can be prompted with text only, video only, or both, without task-specific fine-tuning.","Speaker identity is better preserved: SECS rises to 0.691 (GE2E) and 0.527 (VoxSim) on LRS3 compared with prior video-to-speech methods."],"supporting_citations":[{"why":"MMAudio; supplies a multimodal joint-training baseline and comparison point on VGGSound.","marker":"Cheng et al. (2025)"},{"why":"Faces2Voices; prior video-to-speech state of the art whose WER and SECS numbers the paper beats.","marker":"Kim et al. (2025)"},{"why":"DiffV2S; a diffusion-based video-to-speech baseline for WER comparison.","marker":"Choi et al. (2023a)"},{"why":"Synchformer; provides visual synchronization features and the DeSync metric.","marker":"Iashin et al. (2024)"},{"why":"Kling-Foley; provides the Mel-VAE codec that maps 44.1 kHz audio to 43 Hz latent frames.","marker":"Wang et al. (2025)"},{"why":"F5-TTS; inspiration for the duration-free lyrics-transcription module.","marker":"Chen et al. (2024b)"},{"why":"SD3; provides the MM-DiT block design underlying joint attention.","marker":"Esser et al. (2024)"},{"why":"Whisper; used to transcribe speech for preprocessing and to compute WER on generated speech.","marker":"Radford et al. (2023)"},{"why":"VGGSound; the audio-visual evaluation dataset for audio generation results.","marker":"Chen et al. (2020)"}],"fun_headline_variants":["One model: audio, speech, song synced to video","Unified diffusion transformer for video-synced audio","Video-synced audio, speech, and song from one model","AudioGen-Omni: one model for all video sound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speed and quality claims stand on the assumption that the video-to-speech evaluations feed the model only the silent video, not the ground-truth transcript or lyrics; the paper never states that explicitly, and if the assumption is false the central comparison against other video-to-speech systems is invalid.","fun_headline_variants_meta":{"raw":{"variants":["One model: audio, speech, song synced to video","Unified diffusion transformer for video-synced audio","Video-synced audio, speech, and song from one model","AudioGen-Omni: one model for all video sound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1405,"prompt_tokens":985,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":601,"tokens_out":420,"duration_ms":4231,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:58:26.658384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the model on the LRS3 test set twice: once with only the silent video as conditioning and once with the ground-truth transcript added. If the transcript-free WER is much higher than 17.56 while the transcript-conditioned WER matches it, the claim of visual speech inference collapses; if both are near 17.56, the video-only premise holds.","supporting_citations":[{"cited_title":"Synchformer: Efficient synchronization from sparse cues","cited_arxiv_id":null,"evidence_quote":"Synchformer; provides visual synchronization features and the DeSync metric."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"VGGSound; the audio-visual evaluation dataset for audio generation results."}],"review_version":1}