Pith. sign in

REVIEW 9 cited by

SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.15220 v1 pith:EDWKWCOK submitted 2024-12-03 cs.MM cs.SDeess.AS

Haohe Liu , Gael Le Lan , Xinhao Mei , Zhaoheng Ni , Anurag Kumar , Varun Nagaraja , Wenwu Wang , Mark D. Plumbley
show 2 more authors
Yangyang Shi Vikas Chandra
This is my paper
classification cs.MMcs.SDeess.AS
keywords audiovideosyncflowjointgenerationtextcorrelateddemonstrate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Video and audio are closely correlated modalities that humans naturally perceive together. While recent advancements have enabled the generation of audio or video from text, producing both modalities simultaneously still typically relies on either a cascaded process or multi-modal contrastive encoders. These approaches, however, often lead to suboptimal results due to inherent information losses during inference and conditioning. In this paper, we introduce SyncFlow, a system that is capable of simultaneously generating temporally synchronized audio and video from text. The core of SyncFlow is the proposed dual-diffusion-transformer (d-DiT) architecture, which enables joint video and audio modelling with proper information fusion. To efficiently manage the computational cost of joint audio and video modelling, SyncFlow utilizes a multi-stage training strategy that separates video and audio learning before joint fine-tuning. Our empirical evaluations demonstrate that SyncFlow produces audio and video outputs that are more correlated than baseline methods with significantly enhanced audio quality and audio-visual correspondence. Moreover, we demonstrate strong zero-shot capabilities of SyncFlow, including zero-shot video-to-audio generation and adaptation to novel video resolutions without further training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation

    cs.SD 2025-12 accept novelty 8.0 of 10

    PhyAVBench supplies the first benchmark and contrastive metric that measures whether text-to-audio-video models respect real-world audio physics across controlled prompt pairs.

  2. Inference-Time Scaling for Joint Audio-Video Generation

    cs.MM 2026-06 unverdicted novelty 7.0 of 10

    Presents multi-verifier framework and Adaptive Reward Weighting (ARW) for inference-time scaling in joint audio-video generation, reporting gains in alignment and synchronization on VGGSound and JavisBench-mini.

  3. Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Talker-T2AV achieves better lip-sync accuracy, video quality, and audio quality than dual-branch baselines by separating high-level shared autoregressive modeling from modality-specific low-level diffusion refinement ...

  4. PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation

    cs.SD 2025-12 unverdicted novelty 7.0 of 10

    PhyAVBench provides the first systematic benchmark and metric for audio-physics grounding in T2AV, I2AV, and V2A models using controlled prompt pairs and real video ground truth.

  5. VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    VideoASMR-Bench shows state-of-the-art VLMs fail to reliably detect AI-generated ASMR videos from real ones, though humans can still identify the fakes relatively easily.

  6. Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Unison introduces a unified framework using semantic-guided harmonization and bidirectional cross-modal forcing to generate human-centric videos with improved synchronization between motion, speech, and sound effects.

  7. Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Unison presents a unified audio-video generation model that decouples speech and sound effects while using bidirectional forcing to synchronize with motion, claiming SOTA perceptual quality and alignment.

  8. Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Separating the text condition into a video caption and a visually grounded audio caption, and fusing the diffusion towers with dual cross-attention, gives the reported-best text-to-sounding-video quality and synchroni...

  9. UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A unified audio-video generator built by stitching pre-trained video and music diffusion models, trained on 7,600 hours of data, with a new evaluation benchmark.

Pith tools