Pith. sign in

REVIEW 4 major objections 5 minor 54 references

The paper claims that building cross-modal alignment directly into an audio-video VAE's latent spaces—via segment-level contrastive learning and semantic distillation—makes downstream joint generation better synchronized and higher quality,

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 01:44 UTC pith:2I2B5FEE

load-bearing objection A useful system paper on jointly trained audio–video VAE with semantic distillation and segment-level contrastive alignment, but the ablation does not isolate the effect of the contrastive loss from the longer clips and different training schedule, so the causal claim is weaker than the abstract suggests. the 4 major comments →

arxiv 2607.23855 v2 pith:2I2B5FEE submitted 2026-07-26 cs.SD cs.CV

OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Jun Zhan , Chen Yang , Yitian Gong , Donghua Yu , Kuangwei Chen , Wenbo Zhang , Kexin Huang , Qi Luo
show 18 more authors
Zhe Xu Ying Zhu Jin Wang Tengyue Zhang Qi Chen Cheng Chang Songlin Wang Junqi Dai Jiasheng Ye Xiaogui Yang Tianyi Liang Xiangyu Peng Zhaoye Fei Shimin Li Qinyuan Cheng Xie Chen Xinchi Chen Xipeng Qiu
This is my paper
classification cs.SD cs.CV
keywords audio-video generationcross-modal alignmentcontrastive learningsemantic distillationVAE tokenizerlatent space learnabilitytext-to-audio-video generationsynchronization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a design principle: a tokenizer for joint audio-video generation should not only reconstruct well, but also structure its latent spaces semantically and align them across modalities. OmniVAE is a jointly trained audio-video VAE that adds two training-only objectives to reconstruction: segment-level audio-video contrastive learning, which pulls temporally matching audio and video segments together in a shared embedding space using a bidirectional InfoNCE loss, and per-modality semantic distillation, which injects features from frozen pretrained encoders into each branch. The authors show that these objectives improve the learnability of the latents: a downstream text-to-audio-video model trained on OmniVAE's frozen latents achieves better synchronization, lower word error rate, and better audio quality than models trained on reconstruction-only latents, while reconstruction quality barely drops. If the claim holds, the practical message is that cross-modal correspondence can be paid for once, at tokenizer training time, instead of being re-learned by every downstream generator.

Core claim

The central claim is that fine-grained semantic and temporal correspondence between audio and video can be made an intrinsic property of VAE latent representations rather than something a downstream generative model must infer from scratch. OmniVAE keeps separate audio and video encoders and decoders but trains them jointly with a segment-level bidirectional contrastive objective over one-sixth-second segments, using negatives drawn from the same clip, sibling clips of the same source video, and other videos; this forces the latents to encode temporal-semantic alignment. In parallel, distillation losses push each branch's latents toward the feature geometry of frozen pretrained semantic enco

What carries the argument

The two load-bearing mechanisms are (1) a segment-level audio-video contrastive head that aggregates latent frames into temporally aligned one-sixth-second segment embeddings and applies a bidirectional InfoNCE loss over a hierarchical negative pool, and (2) per-modality semantic distillation projectors that align each VAE latent to features from frozen pretrained semantic encoders via a sigmoid-cosine loss. The contrastive objective carries the cross-modal alignment claim; the distillation objective carries the per-modality learnability claim. Both are training-only: the contrastive head and projectors are discarded at inference, so the deployed tokenizer is just the two VAEs with no extra

Load-bearing premise

The load-bearing premise is that the roughly 23 million training clips are correctly synchronized and semantically paired in fine temporal detail, so the contrastive objective's positives are true positives; if a non-negligible share of clips is misaligned, the model learns false associations and the reported synchronization gains may be inflated by data cleaning.

What would settle it

Train OmniVAE on deliberately shuffled or weakly aligned audio-video pairs under the same objectives; if sync-probing and downstream generation synchronization gains persist, the alignment objective is not the active ingredient. Alternatively, run the frozen OmniVAE latents through a temporal-offset probe on an out-of-distribution dataset with natural audio-video offsets and see whether the fine-grained discrimination survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A downstream text-to-audio-video generator can start from latents that already encode cross-modal correspondence, so it does not have to rediscover synchronization during diffusion training.
  • Video-to-audio systems that currently rely on externally computed synchronization features could instead use the tokenizer's aligned latents, shrinking the gap between training and inference conditioning.
  • The two objectives are complementary: contrastive alignment drives temporal synchronization gains, semantic distillation drives modality-specific quality, and using both yields the strongest aggregate result rather than either alone.
  • Reconstruction quality is largely preserved when the alignment objectives are added, so the approach is compatible with standard VAE-based generation pipelines and can serve as a drop-in replacement tokenizer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The training corpus underwent aggressive filtering for semantic alignment and synchronization, so the measured gains may partly reflect cleaner training data rather than the contrastive objective per se; an ablation on unfiltered pairs would separate the two effects.
  • The claim is evaluated on the same type of concept-filtered web data used for training; a natural next test is whether the alignment transfers to in-the-wild videos with realistic audio-visual offsets and mixed sound sources.
  • The same recipe—distillation plus segment-level contrastive alignment—could generalize to other modality pairs as a strategy for making tokenizers generation-ready, but the paper only demonstrates the audio-video case.
  • The modest sync-probing gain from distillation alone suggests the modality-specific teachers contribute mainly to per-modality quality rather than temporal alignment; synchronization-aware teachers or multi-scale contrastive heads may be a useful extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OmniVAE, a jointly trained audio-video VAE that augments per-modality reconstruction with two training-only objectives: a segment-level bidirectional InfoNCE contrastive loss (Eq. 1) that aligns audio and video latents at fine temporal granularity, and modality-specific semantic distillation from frozen Qwen3-Omni encoders. The authors argue that these objectives make the latent spaces more 'learnable' for downstream generative models, and evaluate this through sync probing on VGS-Sp (§4.3) and by training frozen-VAE text-to-audio-video (T2AV) models (§4.4). The internal ablation compares Recon, Recon+Distill, Recon+AVCLIP, and OmniVAE, reporting improved cross-modal alignment (e.g., DeSync 0.884→0.570, LSE-C 1.479→2.093, WER 0.243→0.168 in Table 4) and better audio quality for OmniVAE. The paper also includes ablations of contrastive pool size/temporal granularity (§4.5.1) and loss-balancing strategies (§4.5.2), and releases code and model weights.

Significance. If the causal claims survive scrutiny, the contribution is valuable: it shows that tokenizer-level, training-only cross-modal alignment can reduce the burden on downstream joint audio-video generators, with no inference overhead. The paper is also commendable for its large-scale 23M-clip corpus, the two complementary objectives, and the evaluation spanning latent probing, reconstruction, and downstream generation. However, the central attribution of the measured gains to the contrastive objective is currently not cleanly identified: the ablation uses unequal training schedules, clip lengths, and initializations across conditions, and the sync-probe hyperparameters are selected on the same benchmark used for headline sync results. The significance is therefore conditional on additional controlled experiments.

major comments (4)
  1. [§4.1, §3.4] The ablation does not isolate the contrastive objective. Recon and Recon+Distill are trained for 250k steps on 121-frame clips independently, while Recon+AVCLIP and OmniVAE train for 82k steps on 193-frame clips with a two-stage freeze/unfreeze schedule, without audio discriminators during joint training, and with an additional decoder fine-tuning stage. Larger temporal context, joint optimization, and the schedule itself are plausible causes of the DeSync/LSE-C/WER gains in Table 4, independent of Eq. (1). A control variant trained jointly with the same clip length and schedule but omitting ℒ_avclip is needed; as written, the headline comparisons cannot be attributed to cross-modal alignment.
  2. [§4.5.1, Table 5] The hyperparameters (FPS, clip duration, segment length) are selected using VGS-Sp A@1, which is the same benchmark used to report the headline sync-probing results in Table 3. This creates a selection-circularity risk; the reported 30.0 A@1 configuration may partly reflect fitting to this evaluation set. Please perform hyperparameter selection on a held-out validation subset and disclose the amount of selection. Also, row '24/—/0.17s' (A@1=7.0) versus '24/2s/0.17s' (A@1=6.0) shows that contrastive training with 2s clips is below the no-contrastive baseline; this non-monotonicity is not discussed and weakens the narrative that larger negative pools uniformly improve alignment.
  3. [§4.3 vs. §3.2/Eq. (1)] The VGS-Sp probe is a temporal-offset classification task, which is essentially the same discrimination the segment-level InfoNCE loss optimizes: the objective already uses intra-clip negatives at nearby temporal positions. Table 3's large A@1 gains (6.4→20.2 frozen) may therefore largely reflect the probe measuring a direct readout of the training objective rather than an independent capability. The paper should rely primarily on the downstream T2AV metrics for the causality claim, or add a probe task that is not a test-time version of the trained objective, and temper the statement that 'contrastive learning provides the main gain in temporal alignment.'
  4. [§4.1, Datasets] The 23M-clip training corpus is filtered using ImageBind semantic alignment, DeSync-based synchronization filtering, and BEATs event detection. If these filters are aggressive, the surviving training distribution is already strongly aligned, so a contrastive objective may preserve filtering-induced alignment rather than create it. Please report filtering retention rates and the distribution of DeSync/alignment scores before and after filtering, or train on an unfiltered (or less filtered) control subset, to show the objective is effective beyond cleaning artifacts.
minor comments (5)
  1. [Abstract] The abstract contains an orphan footnote marker '1' with no corresponding footnote text; please remove it or add the footnote.
  2. [§4.5.2, Table 6] The GradNorm variant is described only as a 'last-layer proxy'; please give a one-sentence explanation or citation for the proxy, since it is not self-explanatory.
  3. [§4.4, Table 4] No confidence intervals, standard errors, or significance tests are reported; given that metrics are averaged over only three checkpoints, please add variance estimates or at least state whether the differences are stable across checkpoints.
  4. [§3.4, Eq. (3)] The text says ℒ_avclip appears in both objectives but is evaluated only once; this is clear in context, but the wording could be tightened to avoid implying two separate contrastive computations.
  5. [§4.5.1, Table 5] The caption 'The dash denotes the reconstruction-only baseline without contrastive training' is ambiguous: it is unclear whether the dash in the Clip column or elsewhere denotes the baseline, and the table is easier to read if the baseline row is explicitly labeled.

Circularity Check

2 steps flagged

Sync probe and segment-retrieval ablation both re-run the InfoNCE objective, but Table 4's external T2AV metrics keep the central claim independently grounded.

specific steps
  1. self definitional [§3.2 Eq. (1)-adjacent text; §4.3 (Table 3)]
    "Inspired by the segment-level contrastive formulation of Synchformer [8], we map the heterogeneous audio and video latents into temporally corresponding segment representations and contrast matched segments against negatives at multiple levels. ... For an anchor segment (i,s), its paired segment in the other modality is the positive, while the negative set N_i,s contains: (i) intra-clip negatives, the other S−1 segments in the same clip; ... We therefore adopt audio-video sync probing as an efficient and sensitive proxy. The probe predicts the temporal offset between audio and video, requiring"

    The InfoNCE loss in Eq. (1) is trained to pick, for each anchor segment, the temporally matched cross-modal segment out of a candidate set whose hardest negatives are the other S−1 segments of the same clip—i.e., segments that differ mainly by temporal offset. The VGS-Sp probe in §4.3 trains a head to classify exactly this kind of temporal offset (audio shifted by N×0.2 s). Therefore the large Table 3 gains for Recon+AVCLIP and OmniVAE are, to a first approximation, a readout of the training objective rather than an independent measure of downstream generation. Because the paper labels the probe a 'proxy' and Table 4 contains external T2AV metrics (Desync, LSE-C, WER, IS, KL), the circularity is partial rather than total.

  2. self definitional [§4.5.2, Table 6]
    "Because all strategies use the same contrastive setup, we evaluate alignment through segment retrieval. Intra A@48 retrieves the paired segment within the same video, whereas Overall A@64 additionally includes negatives sampled from other videos in the batch."

    The retrieval task used here is the same ranking problem as the bidirectional InfoNCE loss in Eq. (1): given an anchor segment, find the paired segment in the other modality among same-clip and cross-video negatives. Table 6 therefore selects the loss-balancing strategy by how well the model performs the training objective itself. This is not an external or downstream measure; it is a self-scoring design criterion. It does not invalidate the downstream Table 4 results, but it is a second instance of evaluating the method with its own training objective.

full rationale

The paper's central claim is that segment-level AVCLIP and semantic distillation make the VAE latents more learnable for downstream text-to-audio-video generation. Table 4 evaluates this with external metrics on generated outputs (Desync, LSE-C, Video-Audio/Text-Audio similarity, IS, KL, FD, WER, AudioBox), which do not reduce to the VAE training losses. The main circularity is §4.3's sync probe and §4.5.2's segment-retrieval ablation: both measure the same temporal-offset discrimination that the InfoNCE objective explicitly trains with intra-clip and cross-video negatives, so their results are partly self-scoring. The paper itself calls the probe a 'proxy', and the downstream results provide independent support, limiting the damage. I also note two non-circular confounds: (1) the joint variants are trained with different schedules (82k steps/193-frame clips, two-stage freeze) than the per-modality baselines (250k steps/121-frame clips), so the causal role of AVCLIP is not fully isolated; (2) the training corpus is filtered with DeSync-based synchronization scoring, and Desync is also a headline evaluation metric, though all compared configurations share the same filtered corpus, so this does not bias the relative ablation. Self-citations (MOSS-Audio-Tokenizer, MOVA) are contextual and not load-bearing. Overall, the derivation is not forced by construction; the circularity burden is moderate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities. Its supported claims rest on a set of modeling and data assumptions: the teacher features are good, the paired data is correctly synchronized, and the downstream generator faithfully reflects latent-space quality. The free parameters are the loss balances and the contrastive/segment hyperparameters, several of which were tuned on the evaluation probe.

free parameters (4)
  • Loss coefficients λ_vr, λ_vd, λ_ar, λ_ad, λ_avclip = Not specified; set by 'loss-magnitude weighting' (§3.4, §4.5.2)
    The balance between reconstruction, distillation, and contrastive losses is hand-set; an ablation of balancing strategies is given, but the chosen constants are not reported numerically.
  • Teacher feature layers = Layer 27 (visual), layer 18 (audio) of Qwen3-Omni (§4.1)
    Chosen without a published layer ablation; the distillation quality depends on this choice.
  • Negative sampling counts and segment granularity = 48 sibling-clip and 24 cross-video negatives; 48 segments of 1/6 s; 24 fps; 8 s clips (§3.2, §4.5.1)
    These hyperparameters were selected using the same VGS-Sp sync-probe set that is later used for the headline sync results (Table 5 vs Table 3), so the main configuration is tuned on the evaluation set.
  • InfoNCE temperature τ = Learned (§3.2)
    A standard learnable temperature; not a fixed constant but a trained parameter of the contrastive loss.
axioms (4)
  • domain assumption Frozen Qwen3-Omni encoders provide semantically meaningful supervision for general video and general audio
    The distillation objective (Eq. 2) supervises VAE latents with these features; if the teacher features are poor for non-speech audio or non-photorealistic video, the reported gains could be teacher-specific.
  • domain assumption The 23M-clip filtering pipeline retains correctly synchronized pairs
    Section 4.1 uses ImageBind scoring, DeSync filtering, and BEATs event detection; false-positive pairs would train the contrastive loss on mismatched segments.
  • domain assumption Downstream T2AV generation on the same team's generator is a faithful probe of latent learnability
    Section 4.4 assumes the flow-matching T2AV model reflects tokenizer quality; adversarial interactions between tokenizer and generator are not controlled.
  • standard math Standard InfoNCE and sigmoid-cosine losses behave as expected with the chosen normalization
    Eqs. 1 and 2 rely on standard properties of InfoNCE and cosine similarity; no novel mathematics is claimed.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation." pith.science (2026). https://pith.science/paper/2I2B5FEE

@misc{pith2026260723855,
  author       = {Pith},
  title        = {Pith review of: OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I2B5FEE}},
  note         = {Machine review of arXiv:2607.23855}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 24 linked inside Pith

  1. [1]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. InInternational Conference on Learning Representations, 2014. URLhttps://arxiv.org/abs/1312.6114

    Pith/arXiv arXiv 2014

  2. [2]

    Reconstructionvs.generation: Tamingoptimizationdilemmainlatent diffusion models

    JingfengYao,BinYang,andXinggangWang. Reconstructionvs.generation: Tamingoptimizationdilemmainlatent diffusion models. arXiv preprint arXiv:2501.01423, 2025. URLhttps://arxiv.org/abs/2501.01423

    Pith/arXiv arXiv 2025

  3. [3]

    Diffusion transformers with representation autoen- coders, 2025

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoen- coders, 2025. URLhttps://arxiv.org/abs/2510.11690

    Pith/arXiv arXiv 2025

  4. [4]

    SpeechTokenizer: Unified speech tokenizer for speech large language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. SpeechTokenizer: Unified speech tokenizer for speech large language models. InInternational Conference on Learning Representations (ICLR) , 2024. URL https: //arxiv.org/abs/2308.16692

    Pith/arXiv arXiv 2024

  5. [5]

    MOSS-Audio-Tokenizer: Scaling audio tokenizers for future audio foundation models, 2026

    OpenMOSS Team. MOSS-Audio-Tokenizer: Scaling audio tokenizers for future audio foundation models, 2026. URL https://arxiv.org/abs/2602.10934

    arXiv 2026

  6. [6]

    MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis

    HoKeiCheng, MasatoIshii, AkioHayakawa, TakashiShibuya, AlexanderSchwing, andYukiMitsufuji. MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 28901–28911, 2025. doi: 10.1109/CVPR52734.2025.02691

    arXiv 2025

  7. [7]

    Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation, 2025

    Sizhe Shan, Qiulin Li, Yutao Cui, Miles Yang, Yuehai Wang, Qun Yang, Jin Zhou, and Zhao Zhong. Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation, 2025. URL https://arxiv.org/abs/2508.16930

    Pith/arXiv arXiv 2025

  8. [8]

    Synchformer: Efficientsynchronizationfromsparse cues

    VladimirIashin,WeidiXie,EsaRahtu,andAndrewZisserman. Synchformer: Efficientsynchronizationfromsparse cues. InICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5325–5329,

    2024

  9. [9]

    Representationlearningwithcontrastivepredictivecoding, 2018

    AaronvandenOord, YazheLi, andOriolVinyals. Representationlearningwithcontrastivepredictivecoding, 2018. URL https://arxiv.org/abs/1807.03748

    Pith/arXiv arXiv 2018

  10. [10]

    Qwen3- Omni technical report, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, et al. Qwen3- Omni technical report, 2025. URLhttps://arxiv.org/abs/2509.17765

    Pith/arXiv arXiv 2025

  11. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning , pages 8748–8763, 2021. URLh...

    Pith/arXiv arXiv 2021

  12. [12]

    Imagebind: One embedding space to bind them all

    RohitGirdhar,AlaaeldinEl-Nouby,ZhuangLiu,MannatSingh,KalyanVasudevAlwala,ArmandJoulin,andIshan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. URLhttps://arxiv.org/abs/2305.05665

    Pith/arXiv arXiv 2023

  13. [13]

    LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment

    BinZhu,BinLin,MunanNing,YangYan,JiaxiCui,HongfaWang,YatianPang,WenhaoJiang,JunwuZhang,Zong- wei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment. InInternational Conference on Learning Representations (ICLR),

  14. [14]

    Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass

    Alexander H. Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. Cross-modal discrete representation learning. InProceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics (ACL), pages 3013–3035, 2022. URLhttps://aclanthology.org/2022.acl-long.215/

    2022

  15. [15]

    URL https://arxiv.org/abs/2310.01852

    Pith/arXiv arXiv

  16. [16]

    Enhancing multimodal unified representations for cross modal generalization

    Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, and Zhou Zhao. Enhancing multimodal unified representations for cross modal generalization. In 12 Findings of the Association for Computational Linguistics (ACL Findings) , 2025. URL https://aclanthology.org/2025. findings-acl.119/

    2025

  17. [17]

    Achieving cross modal generalization with multimodal unified representation

    Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with multimodal unified representation. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. URL https://proceedings. neurips.cc/paper_files/paper/2023/hash/c89f09849eb5af489abb122394ff0f0b-Abstract-Conference.html

    2023

  18. [18]

    Wan: Open and advanced large-scale video generative models, 2025

    Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, et al. Wan: Open and advanced large-scale video generative models, 2025. URLhttps://arxiv.org/abs/2503.20314

    Pith/arXiv arXiv 2025

  19. [19]

    Out of time: Automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: Automated lip sync in the wild. InComputer Vision – ACCV 2016 Workshops, pages 251–263, 2017. doi: 10.1007/978-3-319-54427-4_19

    doi:10.1007/978-3-319-54427-4_19 2016

  20. [20]

    High-fidelity audio compression with improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems , volume 36, pages 27980–27993, 2023. URLhttps://arxiv.org/abs/2306.06546

    Pith/arXiv arXiv 2023

  21. [21]

    Movie gen: A cast of media foundation models, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models, 2024. URLhttps://arxiv. org/abs/2410.13720

    Pith/arXiv arXiv 2024

  22. [22]

    Qwen-audio-vae technical report.arXiv preprint arXiv:2607.11738, 2026

    Ziyue Jiang, Dake Guo, Zekai Zhang, Hangrui Hu, Ting He, Xinfa Zhu, Xiong Wang, Yongqi Wang, Jiapeng Wang, Wenxiang Guo, Zhifang Guo, Chenfei Wu, Dayiheng Liu, and Jin Xu. Qwen-audio-vae technical report.arXiv preprint arXiv:2607.11738, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In Proceedings of the 41st International Conference on Machine Learning , 2024. URLhttps://arxiv.org/abs/2402.04825

    Pith/arXiv arXiv 2024

  24. [24]

    UniTok: A unified tokenizer for visual generation and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. UniTok: A unified tokenizer for visual generation and understanding. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. URLhttps://arxiv.org/abs/2502.20321

    arXiv 2025

  25. [25]

    REPA-E: Unlocking VAEforend-to-endtuningwithlatentdiffusiontransformers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking VAEforend-to-endtuningwithlatentdiffusiontransformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. URLhttps://arxiv.org/abs/2504.10483

    arXiv 2025

  26. [26]

    MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation

    LudanRuan,YiyangMa,HuanYang,HuiguoHe,BeiLiu,JianlongFu,NicholasJingYuan,QinJin,andBainingGuo. MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219–10228, 2023. doi: 10.1109/CVPR52729. 2023.00985

    arXiv 2023

  27. [27]

    Towards scalable pre-training of visual tokenizers for generation, 2025

    Jingfeng Yao, Yuda Song, Yucong Zhou, and Xinggang Wang. Towards scalable pre-training of visual tokenizers for generation, 2025. URLhttps://arxiv.org/abs/2512.13687

    arXiv 2025

  28. [28]

    Sora 2 system card.https://openai.com/index/sora-2-system-card/, 2025

    OpenAI. Sora 2 system card.https://openai.com/index/sora-2-system-card/, 2025. Accessed: 2026-07

    2025

  29. [29]

    AV-DiT: Efficient audio-visual diffusion transformer for joint audio and video generation, 2024

    Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. AV-DiT: Efficient audio-visual diffusion transformer for joint audio and video generation, 2024. URLhttps://arxiv.org/abs/2406.07686

    Pith/arXiv arXiv 2024

  30. [30]

    Seedance 2.0: Advancing video generation for world complexity, 2026

    Seedance Team, ByteDance. Seedance 2.0: Advancing video generation for world complexity, 2026. URLhttps: //arxiv.org/abs/2604.14148

    Pith/arXiv arXiv 2026

  31. [31]

    Veo 3.https://deepmind.google/models/veo/, 2025

    Google DeepMind. Veo 3.https://deepmind.google/models/veo/, 2025. Accessed: 2026-07

    2025

  32. [32]

    Ovi: Twin backbone cross-modal fusion for audio-video genera- tion, 2025

    Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio-video genera- tion, 2025. URLhttps://arxiv.org/abs/2510.01284

    Pith/arXiv arXiv 2025

  33. [33]

    JavisDiT: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchro- nization, 2025

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Jiebo Luo, Ziwei Liu, Hao Fei, and Tat-Seng Chua. JavisDiT: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchro- nization, 2025. URLhttps://arxiv.org/abs/2503.23377

    arXiv 2025

  34. [34]

    MOVA: Towards scalable and synchronized video-audio generation, 2026

    SII-OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, et al. MOVA: Towards scalable and synchronized video-audio generation, 2026. URL https://arxiv.org/abs/2602.08794. 13

    arXiv 2026

  35. [35]

    LTX-2: Efficient joint audio-visual foundation model, 2026

    Yoav HaCohen et al. LTX-2: Efficient joint audio-visual foundation model, 2026. URLhttps://arxiv.org/abs/2601. 03233

    2026

  36. [36]

    Rep- resentation alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Rep- resentation alignment for generation: Training diffusion transformers is easier than you think. InInternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=DJSZGGZYVi

    2025

  37. [37]

    Wan2.2-T2V-A14B model card

    Wan-AI. Wan2.2-T2V-A14B model card. Hugging Face model card, 2025. URLhttps://huggingface.co/Wan-AI/ Wan2.2-T2V-A14B. Accessed: 2026-07-25

    2025

  38. [38]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018. doi: 10.1109/CVPR.2018.00068

    arXiv 2018

  39. [39]

    What matters for representation alignment: Global information or spatial structure?, 2025

    Jaskirat Singh, Xingjian Leng, Zongze Wu, Liang Zheng, Richard Zhang, Eli Shechtman, and Saining Xie. What matters for representation alignment: Global information or spatial structure?, 2025. URLhttps://arxiv.org/abs/ 2512.10794

    arXiv 2025

  40. [40]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 776–780, 2017. doi: 10.1109/ICASSP.2017. 7952261

    doi:10.1109/icassp.2017 2017

  41. [41]

    BEATs: Audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. InProceedings of the 40th International Conference on Machine Learning , pages 5178–5193, 2023. URLhttps://arxiv.org/abs/2212.09058

    Pith/arXiv arXiv 2023

  42. [42]

    UCF101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild, 2012. URLhttps://arxiv.org/abs/1212.0402

    Pith/arXiv arXiv 2012

  43. [43]

    Qwen3.5: Towards native multimodal agents, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. URLhttps://qwen.ai/blog?id=qwen3.5

    2026

  44. [44]

    LibriSpeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: An ASR corpus based on public domain audio books. In2015 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5206–5210, 2015. doi: 10.1109/ICASSP.2015.7178964

    arXiv 2015

  45. [45]

    Panda-70M: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-Wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70M: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13320–13331,...

    arXiv 2024

  46. [46]

    Parker, Matthew Rice, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, Matthew Rice, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio 3,

  47. [47]

    The MUSDB18 corpus for music separation

    Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation. In International Society for Music Information Retrieval Conference, Late-Breaking/Demo Session, 2017. URLhttps://sigsep.github.io/datasets/musdb.html

    2017

  48. [48]

    Sparse in space and time: Audio-visual synchroni- sation with trainable selectors

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Sparse in space and time: Audio-visual synchroni- sation with trainable selectors. InBritish Machine Vision Conference, 2022. URLhttps://arxiv.org/abs/2210.07055

    Pith/arXiv arXiv 2022

  49. [49]

    Metaaudioboxaesthetics: Unifiedautomatic qualityassessmentforspeech,music,andsound

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, MattLe,NickZacharov,CarleighWood,AnnLee,andWei-NingHsu. Metaaudioboxaesthetics: Unifiedautomatic qualityassessmentforspeech,music,andsound. arXiv preprint arXiv:2502.05139,2025. URL https://arxiv.org/abs/ 2502.05139

    Pith/arXiv arXiv 2025

  50. [50]

    VGGSound: A large-scaleaudio-visual dataset

    HonglieChen, WeidiXie, Andrea Vedaldi, and AndrewZisserman. VGGSound: A large-scaleaudio-visual dataset. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 721–725, 2020. doi: 10.1109/ICASSP40776.2020.9053174

    arXiv 2020

  51. [51]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7482–7491, 2018. doi: 10.1109/CVPR.2018.00781. 15

    arXiv 2018

  52. [53]

    UniVerse-1: Unified audio-video generation via stitching of experts, 2025

    Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. UniVerse-1: Unified audio-video generation via stitching of experts, 2025. URLhttps://arxiv.org/abs/ 2509.06155. 14

    Pith/arXiv arXiv 2025

  53. [2024]

    doi: 10.1109/ICASSP48485.2024.10448489

    arXiv 2024

  54. [2026]

    URL https://arxiv.org/abs/2605.17991

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.