Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

JoVA claims that joint video-audio generation with synchronized speech can be achieved by a single joint self-attention mechanism plus a mouth-area loss, without dedicated fusion modules.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:21 UTC pith:NNU5SMCH

load-bearing objection JoVA is a clean architectural simplification with a mouth-region loss that works, but the lip-sync SOTA claims rest on a self-curated 100-sample benchmark and a numerical slip in the text. the 4 major comments →

arxiv 2512.13677 v2 pith:NNU5SMCH submitted 2025-12-15 cs.CV

JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

Xiaohu Huang , Haoyang He , Hao Zhou , Qiangpeng Yang , Shilei Wen , Kai Han This is my paper
classification cs.CV
keywords joint video-audio generationjoint self-attentionlip-syncmouth-area lossdiffusion transformerflow matchingmultimodal generationspeech synthesis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

JoVA claims that joint video-audio generation with synchronized human speech does not require the complex fusion or alignment modules used by current unified models. The paper argues that a single joint self-attention layer letting video, audio, and text tokens interact directly within every transformer block — along with a targeted loss that upweights the mouth region — is enough to reach state-of-the-art lip-sync and speech quality. The authors built a 1.9M-sample training corpus and report that JoVA beats both unified and audio-driven baselines on lip-sync confidence (LSE-C 6.64 vs 6.41) and word error rate (0.18 vs 0.23), even with a 3.2B-parameter variant using far fewer parameters and less training data. If correct, this means architectural simplicity, not added fusion machinery, is a scalable path to human-centric multimodal generation.

Core claim

The paper's central claim is that direct cross-modal interaction via joint self-attention, without dedicated cross-attention or fusion layers, is sufficient — and better — for joint video-audio generation, provided the model gets explicit supervision where lip-speech alignment is hardest. The authors show that when the mouth-area loss weight is zero, lip-sync confidence collapses (LSE-C 1.39), and that raising it to λ=5 lifts LSE-C to 6.64 while leaving audio and video quality metrics roughly unchanged. They further report that their joint self-attention design outperforms two cross-attention variants on lip-sync and word error rate, and that a 3.2B-parameter version of JoVA trained on 1.9M

What carries the argument

Joint self-attention over concatenated video, audio, and text tokens inside each transformer layer, replacing the separate self-attention and cross-attention stacks of prior dual-branch models. Paired with temporal-aligned rotary position embeddings that synchronize video-frame and audio-sample positions, and a mouth-area loss: facial keypoint detection localizes the mouth in pixel space, the bounding box is mapped through the VAE's spatial and temporal downsampling using a sliding-window merge, and the flow-matching objective is applied with extra weight to that masked latent region.

Load-bearing premise

The headline lip-sync and speech results rest on automatic proxies — SyncNet confidence and Whisper WER — measured on a 100-sample benchmark curated by the authors, with no human evaluation or per-sample error analysis; if SyncNet is biased toward the model's output style, the state-of-the-art claim could erode even if the architecture is sound.

What would settle it

A human listening/viewing study where raters judge lip-sync and speech naturalness on a held-out set of diverse talking-head prompts, compared against the same automatic metrics; if human rankings diverge from LSE-C/WER (e.g., OVI ranked higher than JoVA), the paper's central claim fails. Alternatively, running the authors' model on an independent benchmark with more speakers, languages, and head poses would reveal whether the 100-sample result generalizes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Unified multimodal models can drop explicit fusion modules; token concatenation plus shared attention appears sufficient for strong cross-modal alignment.
  • A mouth-area loss is a cheap, architecture-agnostic supervision trick that any talking-head video generator can adopt, yielding large lip-sync gains without hurting other quality metrics.
  • A 3.2B-parameter JoVA trained on 1.9M samples beats a 7.1B-parameter baseline trained on 6.4M samples, suggesting targeted supervision and architecture simplicity can outweigh scale.
  • Low WER (0.18) without a separate text-to-speech stage suggests text-prompt-driven avatar generation can skip cascaded TTS pipelines.
  • The loss weight λ can be tuned across 2–8 with stable audio and video metrics, so the method appears robust to that hyperparameter.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If SyncNet confidence is a faithful proxy, the 1.39→6.50 jump from adding the mouth loss is evidence that generic attention under-weights small high-frequency regions; similar region-weighted losses could help other fine-grained alignment tasks (e.g., eye blinks, finger or instrument movements).
  • The benchmark has only 100 self-curated samples; independent human evaluation and tests on multilingual or accented speech would show whether the LSE-C and WER gains generalize beyond the reported set.
  • The recipe likely extends beyond two modalities: concatenate tokens from N modalities and let shared attention handle all pairs; the mouth-loss principle could become 'difficult-region supervision' for any modality.
  • Initializing the audio branch by duplicating a pretrained video backbone may import video priors, which could explain the slightly lower identity consistency relative to audio-driven baselines; probing this trade-off could guide future initialization choices.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents JoVA, a unified transformer-based framework for joint video-audio generation. The main architectural proposal is to concatenate video and audio tokens and process them with joint self-attention in each block, avoiding separate cross-attention or fusion modules. To improve lip-speech synchronization, the authors add a mouth-area flow-matching loss, computed by mapping detected mouth bounding boxes into VAE latent space via spatial scaling and temporal merging. They construct a training corpus from ambient audio, paired video-audio, and talking-head data, and evaluate on a newly curated 100-sample UniAvatar-Bench and on Verse-Bench. The paper claims state-of-the-art results in lip-sync (LSE-C), speech accuracy (WER), and several audio/video quality metrics compared to joint-generation baselines Universe-1 and OVI, as well as audio-driven models such as Wan-S2V.

Significance. If the claims hold, JoVA provides a genuinely simple and extensible design: joint self-attention removes explicit fusion modules, and the mouth-area loss is a lightweight, architecture-agnostic mechanism for improving lip-sync without modifying the backbone. The ablation studies isolate the contribution of the mouth loss, temporal-aligned RoPE, and joint attention versus cross-attention variants, and the small-model experiment suggests scalability. However, the headline SOTA claim is not fully established: it rests on small margins from a self-curated 100-sample benchmark, uses an automatic SyncNet-based proxy without perceptual validation, contains a numerical inconsistency in the reported LSE-C values, and tunes a key hyperparameter on the same benchmark used for the final claim. The architectural contribution is credible and useful, but the evaluation needs strengthening before the central claim can be accepted.

major comments (4)
  1. [§4.3, Table 2] There is a numerical contradiction on the headline metric. The text states: “Our LSE-C score of 6.51 slightly underperforms OVI (6.41)”, but Table 2 reports OVI’s LSE-C as 6.61. One of these numbers is wrong. Because the SOTA lip-sync claim is built on LSE-C values, this inconsistency must be resolved and all reported numbers re-checked against raw evaluation outputs.
  2. [§4.1, Tables 1 and 3] The SOTA claim rests on UniAvatar-Bench, a self-curated 100-sample benchmark, with no error bars, confidence intervals, significance tests, or per-sample analysis. The margins in Table 1 (LSE-C 6.64 vs 6.43 and 6.41; WER 0.18 vs 0.23) are small and could be within measurement noise. Moreover, the mouth-loss weight λ in Eq. (6) is selected in Table 3 on the same benchmark used to report the headline result, making the final LSE-C a fitted number rather than an independent prediction. Please report uncertainty, perform significance tests, use a held-out validation split for hyperparameter selection, and make the benchmark composition public.
  3. [§4.3, Eq. (7)] The central lip-sync claim is measured only by SyncNet LSE-C, an automatic proxy. Eq. (7) explicitly upweights the mouth region, which can in principle shape the model to maximize statistical alignment that inflates SyncNet confidence without improving human-perceived synchronization. No human evaluation, per-sample error analysis, or correlation with perceptual judgment is provided. I am not claiming this occurs, but the “best lip-sync” claim needs validation against human perception or at least a detailed failure-case analysis before it is established.
  4. [§4.1] UniAvatar-Bench is described only as “100 diverse samples” that are “strictly selected from publicly available datasets” and carefully selected to ensure no overlap with training data. No release statement, source identities, overlap-check details, or selection criteria are provided. Since all comparisons are run on this benchmark, its representativeness and public availability directly affect the validity of the SOTA claim. Please provide the full sample list or URL and detailed curation criteria, or supplement with an existing public benchmark.
minor comments (5)
  1. [§3.2.1, Eq. (4)] Eq. (4) omits the residual connection and feed-forward network that appear in Eq. (1). The notation [h'_v; h'_a; h'_t] = JointAttn([h_v; h_a; h_t]) is ambiguous about whether this is the full transformer block or just the attention sub-layer. Please clarify the exact block computation.
  2. [Title / §1] The title advertises “Generation and Editing”, but the paper contains no editing experiments or editing-specific contributions. Either add editing evaluation or revise the title to match the actual scope.
  3. [Abstract / §4.3 / Table 2] The benchmark name is inconsistent: the abstract says “Universe-Bench” while Section 4.3 and Table 2 use “Verse-Bench”. Also, Table 2 labels the baseline “Universe” while Table 1 labels it “Universe-1”. Please unify naming and citations.
  4. [§3.3] There is a typo: “as seen in see Fig. 3”. Also, the 700-hour talking-head dataset is described only as “collected from publicly available sources”; specifying the sources or a data-cleaning summary would improve reproducibility.
  5. [§4.5] The subsection is titled “Quantitative Results” but contains qualitative comparisons. Please retitle to “Qualitative Results” or add the corresponding quantitative analysis.

Circularity Check

0 steps flagged

No significant circularity; the empirical claims are not derived from their inputs, though the mouth-loss weight is selected on the same benchmark used for the headline table.

full rationale

JoVA is an empirical systems paper rather than a derivation, and no step was found in which a predicted quantity equals an input by construction. The mouth-area loss (Eq. 7) is a training objective whose effect on SyncNet LSE-C is measured, not derived; the architecture ablations (Tables 4 and 5) are controlled comparisons. The central lip-sync claim in Sec. 4.3 is an evaluation result, not a first-principles prediction. The only mild concern is that the mouth-loss weight λ in Eq. (6) is tuned in Table 3, and the selected LSE-C 6.64 is then reported on UniAvatar-Bench in Table 1; if the ablation set is the same as the test set, the headline number is a selection artifact rather than an independent prediction. This is a validation-protocol weakness, not definitional or self-citation circularity: λ does not encode LSE-C, and the loss/metrics are not identified. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling was found; baselines are external and evaluated with official code. Score 2 reflects the minor benchmark-selection concern rather than circular derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

JoVA is an empirical architecture contribution, not a first-principles derivation. It contributes no new physical entities; it inherits Waver/MMAudio/Wan2.1 as external priors. The ledger's main entries are tuned hyperparameters (lambda, CFG) and domain assumptions about VAE locality, metric validity, and data sufficiency.

free parameters (2)
  • Mouth-area loss weight lambda = 5.0
    Selected by ablation on UniAvatar-Bench (Table 3); LSE-C peaks at lambda=5.0. The headline lip-sync number is therefore partly a tuned optimum on the evaluation benchmark.
  • Classifier-free guidance weight = 10.0
    Fixed for both video and audio at inference (Sec. 4.2); no sweep reported, but CFG strongly affects quality metrics and is chosen by hand.
axioms (5)
  • standard math Flow matching (Eq. 2) is a valid generative objective for both video and audio latents.
    Standard diffusion/flow-matching theory invoked in Sec. 3.1; no new derivation.
  • domain assumption Duplicate-initializing the audio branch from the pretrained Waver video backbone transfers useful representations.
    Sec. 3.2.1: 'we initialize an audio diffusion model by duplicating the pretrained video backbone parameters'; no analysis shows this is better than random init or a pretrained audio init.
  • domain assumption Pixel-space mouth bounding boxes map to localized VAE latent regions via scaling and windowed merging.
    Sec. 3.2.2 and App. A; validated only qualitatively on four samples via zero-masking.
  • domain assumption SyncNet LSE-C and Whisper WER are faithful automatic proxies for lip-sync and speech quality on generated content.
    Sec. 4.1 defines these as metrics; all SOTA lip-sync/speech claims rely on them, with no human evaluation.
  • domain assumption The constructed training corpus (2,410h audio + 1,297h paired video-audio + 700h talking-head) is sufficient and representative for the task.
    Sec. 3.3; exact composition, cleaning thresholds, and overlap checks are not released.

pith-pipeline@v1.3.0-alltime-deepseek · 13871 in / 12705 out tokens · 101639 ms · 2026-08-03T16:21:53.854506+00:00 · methodology

0 comments
read the original abstract

In this paper, we present JoVA, a streamlined framework that unifies joint video-audio generation and editing. While existing methods often rely on fragmented, task-specific architectures or complex fusion mechanisms, JoVA employs native joint representation learning for direct video, audio, and text interaction in a dual-branch architecture. This design eliminates redundant alignment modules and effectively unifies diverse multimodal tasks within a single model. Furthermore, we utilize channel-wise conditioning for flexible image and video reference to avoid massive token expansion, alongside a mouth-area loss to enhance lip alignment. To fully empower and systematically evaluate this framework, we construct a comprehensive training corpus encompassing video-audio generation and editing datasets, and introduce unified benchmarks tailored for these multimodal tasks. Extensive experiments demonstrate that JoVA achieves state-of-the-art performance across benchmarks, establishing it as an extensible framework for versatile content creation. Project page: https://visual-ai.github.io/jova

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning

    cs.CV 2026-05 unverdicted novelty 6.0

    SyncDPO improves temporal synchronization in video-audio joint generation using DPO with efficient on-the-fly negative sample construction and curriculum learning.

  2. Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    Mutual Forcing trains a single native autoregressive audio-video model with mutually reinforcing few-step and multi-step modes via self-distillation to match 50-step baselines at 4-8 steps.

  3. OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation

    cs.CV 2026-04 conditional novelty 6.0

    A large-scale, hierarchically-annotated human-centric video dataset and a three-level benchmark (OHBench) are presented, showing that fine-tuning on a subset of the data improves audio-video generation model performance.

  4. OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    OmniHuman is a new large-scale multi-scene dataset with video-, frame-, and individual-level annotations for human-centric video generation, accompanied by the OHBench benchmark that adds metrics aligned with human pe...

  5. OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    OmniShow unifies text, image, audio, and pose conditions into an end-to-end model for high-quality human-object interaction video generation and introduces the HOIVG-Bench benchmark, claiming state-of-the-art results.

  6. Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherence

    cs.CV 2026-04 unverdicted novelty 5.0

    Tora3 uses shared object trajectories as kinematic priors to jointly guide visual motion and acoustic events in audio-video generation, improving realism and synchronization.

  7. From Objectives to Applications: Aligning Architectural Biases in Audio Self-Supervised Learning

    eess.AS 2026-07 unverdicted novelty 3.0

    A survey that organizes audio SSL into five objective paradigms, relates their demands to architectural biases, and interprets downstream applications as tests of generalization.

Reference graph

Works this paper leans on

54 extracted references · 28 linked inside Pith · cited by 6 Pith papers

  1. [1]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters.arXiv preprint arXiv:2505.20156, 2025

    Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu. Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters.arXiv preprint arXiv:2505.20156, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. InCVPR, 2025

    2025

  4. [4]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. InACCV, 2016

    2016

  6. [6]

    Wan2.5.https://wan.video/, 2025

    Alibaba Cloud. Wan2.5.https://wan.video/, 2025

    2025

  7. [7]

    Veo3.https://deepmind.google/models/veo/, 2025

    Google DeepMind. Veo3.https://deepmind.google/models/veo/, 2025

    2025

  8. [8]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. InICASSP, 2020

    2020

  9. [10]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024

    2024

  10. [11]

    Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation.arXiv preprint arXiv:2506.18866, 2025

    Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, and Steven Hoi. Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation.arXiv preprint arXiv:2506.18866, 2025

    Pith/arXiv arXiv 2025

  11. [12]

    Wan-s2v: Audio-driven cinematic video generation.arXiv preprint arXiv:2508.18621, 2025

    Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, et al. Wan-s2v: Audio-driven cinematic video generation.arXiv preprint arXiv:2508.18621, 2025

    Pith/arXiv arXiv 2025

  12. [13]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.arXiv preprint arXiv:2507.08128, 2025

    Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.arXiv preprint arXiv:2507.08128, 2025

    Pith/arXiv arXiv 2025

  13. [14]

    Ace-step: A step towards music generation foundation model

    Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. Ace-step: A step towards music generation foundation model. arXiv preprint arXiv:2506.00045, 2025

    Pith/arXiv arXiv 2025

  14. [15]

    Av-link: Temporally-aligned diffusion features for cross-modal audio-video generation

    Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Alper Canberk, Kwot Sin Lee, Vicente Ordonez, and Sergey Tulyakov. Av-link: Temporally-aligned diffusion features for cross-modal audio-video generation. arXiv preprint arXiv:2412.15191, 2024

    Pith/arXiv arXiv 2024

  15. [16]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  16. [17]

    A simple but strong baseline for sounding video generation: Effective adaptation of audio and video diffusion models for joint generation.arXiv preprint arXiv:2409.17550, 2024

    Masato Ishii, Akio Hayakawa, Takashi Shibuya, and Yuki Mitsufuji. A simple but strong baseline for sounding video generation: Effective adaptation of audio and video diffusion models for joint generation.arXiv preprint arXiv:2409.17550, 2024

    Pith/arXiv arXiv 2024

  17. [18]

    Musiq: Multi-scale image quality transformer

    Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In ICCV, 2021

    2021

  18. [19]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. InACL, 2019

    2019

  19. [20]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition.TASLPRO, 2020

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition.TASLPRO, 2020. 13

    2020

  20. [21]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Pith/arXiv arXiv 2024

  21. [22]

    Efficient training of audio transformers with patchout

    Khaled Koutini, Jan Schlüter, Hamid Eghbal-Zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint arXiv:2110.05069, 2021

    Pith/arXiv arXiv 2021

  22. [23]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    Pith/arXiv arXiv 2025

  23. [24]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  24. [25]

    Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization. arXiv preprint arXiv:2503.23377, 2025

    arXiv 2025

  25. [26]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024

    Pith/arXiv arXiv 2024

  26. [27]

    Ovi: Twin backbone cross-modal fusion for audio-video generation

    Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio-video generation. arXiv preprint arXiv:2510.01284, 2025

    Pith/arXiv arXiv 2025

  27. [28]

    Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.NeurIPS, 2023

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.NeurIPS, 2023

    2023

  28. [29]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. TASLPRO, 2024

    2024

  29. [30]

    Sora2.https://openai.com/zh-Hans-CN/index/sora-2/, 2025

    Openai. Sora2.https://openai.com/zh-Hans-CN/index/sora-2/, 2025

    2025

  30. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023

    2023

  31. [32]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InICML, 2023

    2023

  32. [33]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015

    2015

  33. [34]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. InCVPR, 2023

    2023

  34. [35]

    Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation.arXiv preprint arXiv:2508.16930, 2025

    Sizhe Shan, Qiulin Li, Yutao Cui, Miles Yang, Yuehai Wang, Qun Yang, Jin Zhou, and Zhao Zhong. Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation.arXiv preprint arXiv:2508.16930, 2025

    Pith/arXiv arXiv 2025

  35. [36]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  36. [37]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InECCV, 2020

    2020

  37. [38]

    Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound.arXiv preprint arXiv:2502.05139, 2025

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound.arXiv preprint arXiv:2502.05139, 2025

    Pith/arXiv arXiv 2025

  38. [39]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  39. [40]

    Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025

    Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025. 14

    Pith/arXiv arXiv 2025

  40. [41]

    Kling-foley: Multimodal diffusion transformer for high-quality video-to-audio generation

    Jun Wang, Xijuan Zeng, Chunyu Qiang, Ruilong Chen, Shiyao Wang, Le Wang, Wangjing Zhou, Pengfei Cai, Jiahui Zhao, Nan Li, et al. Kling-foley: Multimodal diffusion transformer for high-quality video-to-audio generation. arXiv preprint arXiv:2506.19774, 2025

    arXiv 2025

  41. [42]

    Av-dit: Taming image diffusion transformers for efficient joint audio and video generation

    Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. Av-dit: Taming image diffusion transformers for efficient joint audio and video generation. InACM Multi, 2025

    2025

  42. [43]

    Audiogen-omni: A unified multimodal diffusion transformer for video-synchronized audio, speech, and song generation.arXiv preprint arXiv:2508.00733, 2025

    Le Wang, Jun Wang, Chunyu Qiang, Feng Deng, Chen Zhang, Di Zhang, and Kun Gai. Audiogen-omni: A unified multimodal diffusion transformer for video-synchronized audio, speech, and song generation.arXiv preprint arXiv:2508.00733, 2025

    Pith/arXiv arXiv 2025

  43. [44]

    Fantasytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. InACM Multi, 2025

    2025

  44. [45]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    Pith/arXiv arXiv 2023

  45. [46]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InICASSP, 2023

    2023

  46. [47]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. InCVPR, pages 7151–7161, 2024

    2024

  47. [48]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv 2024

  48. [49]

    Maniqa: Multi-dimension attention network for no-reference image quality assessment

    Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. InCVPR, 2022

    2022

  49. [50]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  50. [51]

    Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    Pith/arXiv arXiv 2025

  51. [52]

    Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions.arXiv preprint arXiv:2511.03334, 2025

    Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, and Limin Wang. Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions.arXiv preprint arXiv:2511.03334, 2025

    arXiv 2025

  52. [53]

    Waver: Wave your way to lifelike video generation.arXiv preprint arXiv:2508.15761, 2025

    Yifu Zhang, Hao Yang, Yuqi Zhang, Yifei Hu, Fengda Zhu, Chuang Lin, Xiaofeng Mei, Yi Jiang, Bingyue Peng, and Zehuan Yuan. Waver: Wave your way to lifelike video generation.arXiv preprint arXiv:2508.15761, 2025

    Pith/arXiv arXiv 2025

  53. [54]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds.arXiv preprint arXiv:2407.01494, 2024

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds.arXiv preprint arXiv:2407.01494, 2024

    Pith/arXiv arXiv 2024

  54. [55]

    Uniform: A unified diffusion transformer for audio-video generation.arXiv e-prints, 2025

    Lei Zhao, Linfeng Feng, Dongxu Ge, Fangqiu Yi, Chi Zhang, Xiao-Lei Zhang, and Xuelong Li. Uniform: A unified diffusion transformer for audio-video generation.arXiv e-prints, 2025. 15 Appendix A Mouth-Area Mask Validation LatentSpaceReconstructedClip VA EEncoder VA EDecoder KeypointDetection Down-sample Mask out features RawClip Figure 6 Mouth-area mask ge...

    2025