Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Jointly training an audio-video VAE with segment contrast and semantic distillation makes synchronized generation easier for the model that follows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 13:54 UTC pith:2I2B5FEE

load-bearing objection Solid empirical methods paper: segment-level A/V contrastive + semantic distillation inside a dual generative VAE clearly helps sync and quality; the “full combo is strongest” claim is a bit thinner than the abstract sells. the 4 major comments →

arxiv 2607.23855 v1 pith:2I2B5FEE submitted 2026-07-26 cs.SD cs.CV

OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Jun Zhan , Chen Yang , Yitian Gong , Donghua Yu , Kuangwei Chen , Wenbo Zhang , Kexin Huang , Qi Luo
show 18 more authors
Zhe Xu Ying Zhu Jin Wang Tengyue Zhang Qi Chen Cheng Chang Songlin Wang Junqi Dai Jiasheng Ye Xiaogui Yang Tianyi Liang Xiangyu Peng Zhaoye Fei Shimin Li Qinyuan Cheng Xie Chen Xinchi Chen Xipeng Qiu
This is my paper
classification cs.SD cs.CV
keywords audio-video VAEcross-modal alignmentsegment-level contrastive learningsemantic distillationtext-to-audio-video generationlatent space learnabilitysynchronization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most joint audio-video generators still compress sound and picture with separately trained VAEs, so the generator must learn both the data and the correspondence from scratch. OmniVAE instead trains the two VAEs together: a segment-level contrastive loss aligns audio and video latents at fine temporal grain, and distillation from frozen modality-specific encoders injects semantic structure into each space. Both objectives are training-only and add no inference cost. Probing and full text-to-audio-video runs show that each objective helps on its own and that combining them yields the strongest cross-modal sync and generation quality while keeping reconstruction competitive. The claim is that unified, aligned latents are a better foundation for omnimodal generation than reconstruction-only tokenizers.

Core claim

Embedding modality-specific semantics and fine-grained audio-video correspondence directly into the VAE latent spaces improves how learnable those spaces are for downstream generators, producing higher generation quality and more accurate temporal-semantic synchronization than reconstruction-only or single-objective variants, without material loss of reconstruction fidelity.

What carries the argument

OmniVAE: separate audio and video VAEs jointly trained with a segment-level bidirectional InfoNCE alignment loss (hierarchical negatives over clip, sibling clip, and cross-video segments) plus per-modality semantic distillation from frozen encoders; both heads are discarded at inference.

Load-bearing premise

That the measured gains will hold outside the authors’ own filtered training corpus, rewritten evaluation prompts, and single twin-tower flow-matching generator recipe.

What would settle it

Train the same text-to-audio-video generator on OmniVAE versus a matched reconstruction-only VAE pair using an independent public audio-visual corpus and unaltered public lip-sync and desync benchmarks; if DeSync, lip-sync, and audio quality no longer improve under OmniVAE, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Downstream joint generators can spend capacity on content rather than inventing cross-modal sync from misaligned latents.
  • Tokenizer design for omnimodal models should treat alignment and semantics as first-class training goals, not post-hoc generator tricks.
  • Training-only contrastive and distillation heads can upgrade existing separate VAEs without changing the inference interface.
  • Stronger segment-level negatives and finer temporal grain should further tighten measurable audio-video sync in generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same recipe may transfer to other modality pairs that share time (e.g., video-text motion, audio-IMU) where separate autoencoders currently leave alignment to the generator.
  • If aligned latents reduce the need for external sync features, twin-tower generators could drop or shrink dedicated synchronization conditioners.
  • A natural next test is whether OmniVAE latents improve frozen retrieval or offset classification on public sets without any generator training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents OmniVAE, a jointly trained audio-video VAE pair (Wan2.2-based video branch, DAC-style audio branch) that augments per-modality reconstruction with two training-only objectives: a segment-level bidirectional InfoNCE loss (AVCLIP) with a hierarchical source-video/clip/segment negative pool, and modality-specific feature distillation from frozen Qwen3-Omni encoders. A progressive ablation (Recon / +Distill / +AVCLIP / full) evaluates reconstruction (Table 2), a sync-offset probe on VGS-Sp (Table 3), and downstream text-to-audio-video generation on Verse-Bench with a twin-tower flow-matching model (Table 4, Fig. 2). The headline findings are that AVCLIP substantially improves cross-modal alignment (sync probe A@1 6.4→18.1 frozen; DeSync 0.884→0.576), distillation improves audio quality metrics, and their combination gives the best overall downstream results, with only modest reconstruction cost. Ablations cover contrastive temporal granularity (Table 5) and loss-balancing strategy (Table 6).

Significance. If the results hold, this is a useful contribution: it is, to my knowledge, the first audio-video tokenizer explicitly trained for cross-modal latent alignment, and the design is practical — both auxiliary objectives are training-only, adding zero inference cost (Table 1), which makes adoption by downstream generators frictionless. The ablation design is commendably clean: the sync probe is a held-out, inexpensive proxy, and the large AVCLIP effects (DeSync 0.884→0.576; probe A@1 6.4→18.1) are far outside plausible noise. The evaluation uses external metrics throughout (SyncNet/DeSync, AudioBox, IS/KL/FD/WER) rather than redefined training objectives. Code, model weights, and a project page are released, supporting reproducibility. The main limitation on significance is that all downstream evidence comes from a single in-house data corpus, benchmark adaptation (LLM-rewritten Set3 prompts), and generator recipe, so transfer to independent stacks is untested.

major comments (4)
  1. [§4.4, Table 4] Table 4 reports means over the final three checkpoints of a single training run. The claim that the full model is strongest rests partly on small deltas — DeSync 0.576→0.570 (~1%) and LSE-C 1.970→2.093 — and §4.4 states LSE-C is computable only on the face-detectable fraction of 100 rewritten Set3 prompts, without reporting that n. The per-component claims (AVCLIP helps, distillation helps) are supported by large margins, but 'combination is strongest' is not established against seed-to-seed variance. Please report the Set3 face-detectable n, and either add a second seed for the key alignment metrics or explicitly qualify the full-model claim.
  2. [§4.4, Table 4 (Video Quality)] The Motion Score column shows a monotone, substantial degradation across configurations: 0.556 (Recon) → 0.495 (+Distill) → 0.533 (+AVCLIP) → 0.456 (OmniVAE), an 18% relative drop for the full model. The text (§4.4) describes motion quality as 'broadly comparable across configurations.' This is a mismatch between data and claim; it also complicates attribution, since distillation alone already costs ~11%. Either the metric threshold for 'comparable' should be justified, or the text should acknowledge the regression and discuss whether it reflects a real dynamics cost of the semantic objectives.
  3. [§3.4 / §4.1, Table 2(b)] The ablation is not fully training-matched on the audio side. §4.1 states that only the AVCLIP-based variants undergo the Stage-3 audio-decoder adversarial refinement (to remove artifacts from joint training), while Recon and Recon+Distill retain discriminators throughout. Since the decoder maps generated latents to waveforms, this asymmetric refinement could advantage AVCLIP variants on the audio metrics in Tables 2(b) and 4. Please clarify whether Table 2(b) is measured pre- or post-refinement for all variants, and ideally report the AVCLIP variants pre-refinement so the comparison isolates the objective rather than the pipeline.
  4. [§4.1, §4.4 (data filtering vs. metric)] The training corpus is filtered with DeSync-based synchronization filtering (§4.1), and DeSync is then the headline alignment metric in Table 4. Because all four variants share the data, the ablation ordering is not confounded, but absolute DeSync values are likely optimistic relative to unfiltered distributions. A short evaluation on an external, unfiltered audio-visual set (or an explicit statement of this limitation) would strengthen the generalization claim; currently all evidence lives inside one data/benchmark/generator stack.
minor comments (6)
  1. [§4.5.1, Table 5] Table 5 is hard to parse: the row '24 — 0.17s / 7.0' uses a dash for clip duration, and the caption says the dash denotes the reconstruction-only baseline, which does not match the row's contrastive-looking entries. Please reformat so each row is unambiguous.
  2. [§4.2, Table 2(b)] In Table 2(b), AVCLIP degrades audio reconstruction more than the text's 'slight degradation' suggests (SIM 0.9893→0.9666, ViSQOL 4.55→4.03 on AudioSet). A sentence quantifying the cost, and whether Stage-3 refinement closes it, would help.
  3. [§4.4, Fig. 2] Fig. 2 legend uses internal run names (recon_distill_avclip) while the tables use 'OmniVAE'; please unify. Also consider stating the y-axis scales explicitly, since several panels are zoomed in ways that visually amplify small differences.
  4. [§1, §2.2] The 'first audio-video VAE designed for cross-modal latent alignment' claim appears twice (§1, §2.2). Once suffices, and given concurrent twin-tower systems (Ovi, LTX-2, MOVA) a slightly softer phrasing ('to the best of our knowledge') is already present — keep it consistent in both places.
  5. [§3.4 / §4.1] Sec. 4.1 does not state the values of the loss coefficients λ or the learned temperature τ, despite these being central to the balancing discussion in §4.5.2. Please report them (or their magnitudes) for reproducibility, alongside the 2:2:1 negative ratio already given.
  6. [§3.5] The downstream T2AV model (§3.5) is described only at a high level; parameter count of the cross-attention bridges, training steps, and CFG setting are scattered (CFG=5 appears only in §4.4). A compact summary of the generator configuration would make Table 4 self-contained.

Simulated Author's Rebuttal

4 responses · 2 unresolved

We thank the referee for the careful reading and the overall positive assessment, particularly the recognition that the per-component claims are supported by large margins and that the training-only design makes adoption frictionless. We agree with the substance of all four major comments and will revise accordingly: we will report the Set3 face-detectable n and qualify the "combination is strongest" claim (and add a second seed for the key alignment metrics where compute permits); we will correct the motion-quality description in §4.4 to acknowledge the Motion Score regression and discuss its likely cause; we will clarify the pre/post-refinement status of Table 2(b) and add pre-refinement numbers for the AVCLIP variants; and we will add an explicit limitation on DeSync-based data filtering versus DeSync as an evaluation metric, together with a small external-set sync-probe evaluation. Detailed responses follow.

read point-by-point responses

  1. Referee: Table 4: single training run, small deltas for the full-model claim (DeSync 0.576→0.570, LSE-C 1.970→2.093), and the Set3 face-detectable n is not reported. Report n, and either add a second seed for key alignment metrics or explicitly qualify the full-model claim.

    Authors: We agree. In the revision we will (1) report the exact face-detectable n for the rewritten Set3 prompts alongside the LSE-C column, and mark LSE-C explicitly as computed on that subset; (2) retrain the T2AV model for the two key configurations (Recon+AVCLIP and OmniVAE) with a second seed and report mean ± spread for DeSync and the sync-probe A@1, which are the metrics underlying the alignment claim. Given the cost of full T2AV retraining, if the second-seed runs cannot be completed within the revision window we will instead qualify the claim in §4.4 and the abstract: we will state that AVCLIP's benefit over non-AVCLIP variants is large and robust (DeSync 0.884→0.576, probe A@1 6.4→18.1), while the incremental gain of adding distillation on top of AVCLIP is small (~1% on DeSync) and within plausible seed variance, and should be read as 'no degradation plus consistent small improvements' rather than a decisive win. We will also report per-checkpoint values (180k/190k/200k) rather than only their mean, so readers can see the checkpoint-to-checkpoint spread. revision: partial

  2. Referee: Motion Score degrades monotonically and substantially (0.556→0.495→0.533→0.456, −18% for the full model), but §4.4 calls motion quality 'broadly comparable.' Justify the threshold or acknowledge the regression and discuss whether it reflects a real dynamics cost of the semantic objectives.

    Authors: The referee is correct; 'broadly comparable' does not accurately describe a monotone 18% relative drop, and we will revise the text. Our current understanding, which we will present as a discussion point rather than a settled conclusion: distillation alone already costs ~11% (0.556→0.495), suggesting the effect is primarily associated with the distillation objective rather than AVCLIP; one plausible mechanism is that aligning latents to frozen semantic-encoder features biases the latent space toward per-frame semantic content at some cost to fine temporal dynamics, which the downstream generator then reproduces as reduced motion magnitude. We note, however, that Motion Score (a flow-magnitude-based metric) conflates motion amount with motion quality, and lower scores can partly reflect reduced high-frequency jitter; we will state this caveat explicitly and avoid claiming equivalence. Concretely, the revision will (1) replace 'broadly comparable' with an accurate description of the regression, (2) attribute it mainly to distillation based on the +Distill row, (3) discuss the mechanism and the metric's limitations, and (4) list mitigation (e.g., reduced distillation weight or later-stage distillation scheduling) as future work. revision: yes

  3. Referee: Asymmetric audio-decoder refinement: only AVCLIP variants undergo Stage-3 adversarial refinement while Recon/Recon+Distill retain discriminators throughout. Clarify whether Table 2(b) is pre- or post-refinement for all variants, and ideally report AVCLIP variants pre-refinement to isolate the objective from the pipeline.

    Authors: We apologize for the ambiguity. Table 2(b) as currently reported is post-refinement for the AVCLIP-based variants and reflects the final model a user would deploy; we will state this explicitly in §4.1 and the Table 2 caption. To isolate the objective from the pipeline, we will add pre-refinement rows for Recon+AVCLIP and OmniVAE to Table 2(b). We note for context that the refinement freezes the audio encoder and fine-tunes only the decoder, so the latent space — and hence all alignment metrics (Table 3) and the latents consumed by the downstream generator — are unaffected; the refinement can only influence waveform-level audio metrics. Our internal logs indicate the pre-refinement gap is concentrated in the perceptual metrics (ViSQOL, PESQ), consistent with the removal of electronic artifacts, while mel/STFT distances change little; the added rows will let readers verify this directly. We will also add a sentence clarifying that Recon and Recon+Distill never lost adversarial supervision, so no refinement was needed for them. revision: yes

  4. Referee: Training data is filtered with DeSync-based synchronization filtering, and DeSync is then the headline alignment metric. Ablation ordering is not confounded, but absolute DeSync values are likely optimistic. Add an evaluation on an external unfiltered audio-visual set, or explicitly state the limitation; all evidence currently lives in one data/benchmark/generator stack.

    Authors: We agree on both points. First, the revision will include an explicit limitation statement: because the training corpus is DeSync-filtered, absolute DeSync values in Table 4 are measured on a distribution closer to the training data than an unfiltered web distribution, and should be interpreted as relative comparisons across variants (which share the data) rather than absolute synchronization quality. Second, we will add an evaluation on an external, unfiltered audio-visual set: we will run the frozen sync probe (Table 3 protocol) on a held-out, unfiltered web-video sample disjoint from our corpus and report A@1 for all four variants, testing whether the AVCLIP ordering and margin hold off-distribution. We note that VGS-Sp is already external to our training corpus, so the probe results in Table 3 are not affected by the filtering concern; the new experiment addresses the DeSync-metric side specifically. Transfer to an independent generator stack remains untested and we will state this plainly in the conclusion as a limitation and direction for future work, rather than claim broader generality. revision: partial

standing simulated objections not resolved
  • A second T2AV seed for the key alignment metrics may not be completable within the revision window due to training cost; if so, the authors commit only to qualifying the full-model claim and reporting per-checkpoint spread, which addresses the comment only partially.
  • Transfer of the approach to an independent data/benchmark/generator stack cannot be demonstrated within this revision and remains an acknowledged open limitation.

Circularity Check

0 steps flagged

No significant circularity: training objectives are distinct from evaluation metrics, and gains are measured on held-out probes and a separately trained T2AV generator.

full rationale

OmniVAE’s load-bearing claims are empirical ablations, not first-principles derivations. The segment-level bidirectional InfoNCE (Eq. 1) and sigmoid-cosine distillation (Eq. 2) are training-only losses; they are not redefined as the reported outcomes. Alignment is probed with a separate temporal-offset classifier on VGGSound-Sparse (Table 3), and generation quality/sync are measured after freezing the VAE and training an independent flow-matching T2AV model, using external metrics (DeSync, LSE-C, IS, KL, FD, WER, AudioBox, aesthetic predictors; Table 4, Fig. 2). Reconstruction is checked with standard PSNR/SSIM/LPIPS and speech/audio metrics (Table 2). Citations to Synchformer, Qwen3-Omni, Wan-VAE, and related OpenMOSS work supply architectural priors or teachers, not uniqueness theorems that force the result. Shared DeSync-based corpus filtering affects all four VAE variants equally and does not make the ablation ordering true by construction. No step reduces a claimed prediction to a fitted input or to a self-definitional identity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The work is empirical ML. It rests on standard VAE/contrastive/distillation machinery, pretrained backbones and teachers, and engineering choices (segment length, negative hierarchy, loss-magnitude weights, three-stage schedule). No new physical entities. Free parameters are the usual loss weights, temperatures, and pool sizes selected by ablation. Domain assumptions include that Qwen3-Omni features are good semantic teachers and that the filtered web corpus plus AudioSet supplies genuine fine-grained A/V correspondence.

free parameters (4)
  • loss coefficients λ_vr, λ_vd, λ_ar, λ_ad, λ_avclip = magnitude-balanced (contrastive-anchored)
    Set by loss-magnitude weighting so reconstruction, distillation, and contrastive terms have comparable scale; chosen via ablation (Table 6), not derived.
  • InfoNCE temperature τ = learnable
    Learnable temperature in the bidirectional segment InfoNCE; standard free parameter of contrastive losses.
  • segment count S=48, clip duration 8s, segment length ~0.17s, negative ratio ~2:2:1 = S=48, 8s clips, 47+48+24 negatives
    Hierarchical negative construction and temporal granularity selected by sync-probe ablation (Table 5); directly affects alignment strength.
  • distillation teacher layers (vision L27, audio L18) = L27 vision / L18 audio
    Which Qwen3-Omni layers supply targets is a design choice affecting semantic content of latents.
axioms (5)
  • domain assumption Standard VAE evidence lower bound plus adversarial/feature-matching reconstruction yields a usable continuous latent for diffusion/flow generators.
    Assumed throughout Sec. 3; backbones are Wan2.2 and DAC-style VAEs pretrained with reconstruction.
  • domain assumption Bidirectional segment-level InfoNCE on aggregated latents induces fine-grained temporal-semantic A/V correspondence useful to downstream generators.
    Core methodological bet of Sec. 3.2, imported from Synchformer-style discriminative sync learning and applied to generative latents.
  • domain assumption Frozen Qwen3-Omni vision/audio encoder features are sufficiently strong semantic teachers that cosine distillation improves latent learnability.
    Sec. 3.3; follows VA-VAE/REPA/iREPA line without independent proof that these layers are optimal for generative VAEs.
  • ad hoc to paper The custom ~23M filtered speech/non-speech corpus plus AudioSet is representative enough that measured T2AV gains will matter outside this stack.
    Sec. 4.1 data pipeline (LLM concept tree, ImageBind-style scoring, DeSync filter, BEATs AED) is paper-specific; generalization is assumed.
  • ad hoc to paper Loss-magnitude weighting keeps multi-task gradients well-behaved relative to GradNorm or uncertainty weighting.
    Chosen after Table 6 ablation; not a general theorem.
invented entities (2)
  • OmniVAE (jointly trained dual audio-video VAE with training-only contrastive head and distillation projectors) independent evidence
    purpose: Provide decodable, modality-specific latents that already encode segment-level cross-modal alignment and semantic structure for downstream joint generation.
    New system-level artifact; not a new particle or force, but the named object whose properties are claimed. Independent evidence is the released weights and the reported probe/T2AV metrics.
  • Hierarchical source-video–clip–segment negative pool (intra-clip / sibling-clip / cross-video) no independent evidence
    purpose: Supply hard temporal negatives plus semantic diversity for segment InfoNCE beyond Synchformer's pool.
    Engineering construct introduced in Sec. 3.2; validated only inside this training setup.

pith-pipeline@v1.2.0-grok45-kimik3 · 20222 in / 3560 out tokens · 68457 ms · 2026-07-30T13:54:24.956403+00:00 · methodology

0 comments
read the original abstract

Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 1 canonical work pages

  1. [1]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. InInternational Conference on Learning Representations, 2014. URLhttps://arxiv.org/abs/1312.6114

    Pith/arXiv arXiv 2014

  2. [2]

    Reconstructionvs.generation: Tamingoptimizationdilemmainlatent diffusion models

    JingfengYao,BinYang,andXinggangWang. Reconstructionvs.generation: Tamingoptimizationdilemmainlatent diffusion models. arXiv preprint arXiv:2501.01423, 2025. URLhttps://arxiv.org/abs/2501.01423

    Pith/arXiv arXiv 2025

  3. [3]

    Diffusion transformers with representation autoen- coders, 2025

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoen- coders, 2025. URLhttps://arxiv.org/abs/2510.11690

    Pith/arXiv arXiv 2025

  4. [4]

    SpeechTokenizer: Unified speech tokenizer for speech large language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. SpeechTokenizer: Unified speech tokenizer for speech large language models. InInternational Conference on Learning Representations (ICLR) , 2024. URL https: //arxiv.org/abs/2308.16692

    Pith/arXiv arXiv 2024

  5. [5]

    MOSS-Audio-Tokenizer: Scaling audio tokenizers for future audio foundation models, 2026

    OpenMOSS Team. MOSS-Audio-Tokenizer: Scaling audio tokenizers for future audio foundation models, 2026. URL https://arxiv.org/abs/2602.10934

    arXiv 2026

  6. [6]

    MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis

    HoKeiCheng, MasatoIshii, AkioHayakawa, TakashiShibuya, AlexanderSchwing, andYukiMitsufuji. MMAudio: Taming multimodal joint training for high-quality video-to-audio synthesis. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 28901–28911, 2025. doi: 10.1109/CVPR52734.2025.02691

    arXiv 2025

  7. [7]

    Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation, 2025

    Sizhe Shan, Qiulin Li, Yutao Cui, Miles Yang, Yuehai Wang, Qun Yang, Jin Zhou, and Zhao Zhong. Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation, 2025. URL https://arxiv.org/abs/2508.16930

    Pith/arXiv arXiv 2025

  8. [8]

    Synchformer: Efficientsynchronizationfromsparse cues

    VladimirIashin,WeidiXie,EsaRahtu,andAndrewZisserman. Synchformer: Efficientsynchronizationfromsparse cues. InICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5325–5329,

    2024

  9. [9]

    Representationlearningwithcontrastivepredictivecoding, 2018

    AaronvandenOord, YazheLi, andOriolVinyals. Representationlearningwithcontrastivepredictivecoding, 2018. URL https://arxiv.org/abs/1807.03748

    Pith/arXiv arXiv 2018

  10. [10]

    Qwen3- Omni technical report, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, et al. Qwen3- Omni technical report, 2025. URLhttps://arxiv.org/abs/2509.17765

    Pith/arXiv arXiv 2025

  11. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning , pages 8748–8763, 2021. URLh...

    Pith/arXiv arXiv 2021

  12. [12]

    Imagebind: One embedding space to bind them all

    RohitGirdhar,AlaaeldinEl-Nouby,ZhuangLiu,MannatSingh,KalyanVasudevAlwala,ArmandJoulin,andIshan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. URLhttps://arxiv.org/abs/2305.05665

    Pith/arXiv arXiv 2023

  13. [13]

    LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment

    BinZhu,BinLin,MunanNing,YangYan,JiaxiCui,HongfaWang,YatianPang,WenhaoJiang,JunwuZhang,Zong- wei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. LanguageBind: Extending video-language pretraining to n-modality by language-based semantic alignment. InInternational Conference on Learning Representations (ICLR),

  14. [14]

    Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass

    Alexander H. Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. Cross-modal discrete representation learning. InProceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics (ACL), pages 3013–3035, 2022. URLhttps://aclanthology.org/2022.acl-long.215/

    2022

  15. [15]

    URL https://arxiv.org/abs/2310.01852

    Pith/arXiv arXiv

  16. [16]

    Enhancing multimodal unified representations for cross modal generalization

    Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, and Zhou Zhao. Enhancing multimodal unified representations for cross modal generalization. In 12 Findings of the Association for Computational Linguistics (ACL Findings) , 2025. URL https://aclanthology.org/2025. findings-acl.119/

    2025

  17. [17]

    Achieving cross modal generalization with multimodal unified representation

    Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with multimodal unified representation. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. URL https://proceedings. neurips.cc/paper_files/paper/2023/hash/c89f09849eb5af489abb122394ff0f0b-Abstract-Conference.html

    2023

  18. [18]

    Wan: Open and advanced large-scale video generative models, 2025

    Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, et al. Wan: Open and advanced large-scale video generative models, 2025. URLhttps://arxiv.org/abs/2503.20314

    Pith/arXiv arXiv 2025

  19. [19]

    Out of time: Automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: Automated lip sync in the wild. InComputer Vision – ACCV 2016 Workshops, pages 251–263, 2017. doi: 10.1007/978-3-319-54427-4_19

    work page doi:10.1007/978-3-319-54427-4_19 2016

  20. [20]

    High-fidelity audio compression with improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems , volume 36, pages 27980–27993, 2023. URLhttps://arxiv.org/abs/2306.06546

    Pith/arXiv arXiv 2023

  21. [21]

    Movie gen: A cast of media foundation models, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models, 2024. URLhttps://arxiv. org/abs/2410.13720

    Pith/arXiv arXiv 2024

  22. [22]

    Qwen-audio-vae technical report.arXiv preprint arXiv:2607.11738, 2026

    Ziyue Jiang, Dake Guo, Zekai Zhang, Hangrui Hu, Ting He, Xinfa Zhu, Xiong Wang, Yongqi Wang, Jiapeng Wang, Wenxiang Guo, Zhifang Guo, Chenfei Wu, Dayiheng Liu, and Jin Xu. Qwen-audio-vae technical report.arXiv preprint arXiv:2607.11738, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In Proceedings of the 41st International Conference on Machine Learning , 2024. URLhttps://arxiv.org/abs/2402.04825

    Pith/arXiv arXiv 2024

  24. [24]

    UniTok: A unified tokenizer for visual generation and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. UniTok: A unified tokenizer for visual generation and understanding. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. URLhttps://arxiv.org/abs/2502.20321

    arXiv 2025

  25. [25]

    REPA-E: Unlocking VAEforend-to-endtuningwithlatentdiffusiontransformers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking VAEforend-to-endtuningwithlatentdiffusiontransformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. URLhttps://arxiv.org/abs/2504.10483

    arXiv 2025

  26. [26]

    MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation

    LudanRuan,YiyangMa,HuanYang,HuiguoHe,BeiLiu,JianlongFu,NicholasJingYuan,QinJin,andBainingGuo. MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219–10228, 2023. doi: 10.1109/CVPR52729. 2023.00985

    arXiv 2023

  27. [27]

    Towards scalable pre-training of visual tokenizers for generation, 2025

    Jingfeng Yao, Yuda Song, Yucong Zhou, and Xinggang Wang. Towards scalable pre-training of visual tokenizers for generation, 2025. URLhttps://arxiv.org/abs/2512.13687

    arXiv 2025

  28. [28]

    Sora 2 system card.https://openai.com/index/sora-2-system-card/, 2025

    OpenAI. Sora 2 system card.https://openai.com/index/sora-2-system-card/, 2025. Accessed: 2026-07

    2025

  29. [29]

    AV-DiT: Efficient audio-visual diffusion transformer for joint audio and video generation, 2024

    Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. AV-DiT: Efficient audio-visual diffusion transformer for joint audio and video generation, 2024. URLhttps://arxiv.org/abs/2406.07686

    Pith/arXiv arXiv 2024

  30. [30]

    Seedance 2.0: Advancing video generation for world complexity, 2026

    Seedance Team, ByteDance. Seedance 2.0: Advancing video generation for world complexity, 2026. URLhttps: //arxiv.org/abs/2604.14148

    Pith/arXiv arXiv 2026

  31. [31]

    Veo 3.https://deepmind.google/models/veo/, 2025

    Google DeepMind. Veo 3.https://deepmind.google/models/veo/, 2025. Accessed: 2026-07

    2025

  32. [32]

    Ovi: Twin backbone cross-modal fusion for audio-video genera- tion, 2025

    Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio-video genera- tion, 2025. URLhttps://arxiv.org/abs/2510.01284

    Pith/arXiv arXiv 2025

  33. [33]

    JavisDiT: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchro- nization, 2025

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Jiebo Luo, Ziwei Liu, Hao Fei, and Tat-Seng Chua. JavisDiT: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchro- nization, 2025. URLhttps://arxiv.org/abs/2503.23377

    arXiv 2025

  34. [34]

    MOVA: Towards scalable and synchronized video-audio generation, 2026

    SII-OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, et al. MOVA: Towards scalable and synchronized video-audio generation, 2026. URL https://arxiv.org/abs/2602.08794. 13

    arXiv 2026

  35. [35]

    LTX-2: Efficient joint audio-visual foundation model, 2026

    Yoav HaCohen et al. LTX-2: Efficient joint audio-visual foundation model, 2026. URLhttps://arxiv.org/abs/2601. 03233

    2026

  36. [36]

    Rep- resentation alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Rep- resentation alignment for generation: Training diffusion transformers is easier than you think. InInternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=DJSZGGZYVi

    2025

  37. [37]

    Wan2.2-T2V-A14B model card

    Wan-AI. Wan2.2-T2V-A14B model card. Hugging Face model card, 2025. URLhttps://huggingface.co/Wan-AI/ Wan2.2-T2V-A14B. Accessed: 2026-07-25

    2025

  38. [38]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018. doi: 10.1109/CVPR.2018.00068

    arXiv 2018

  39. [39]

    What matters for representation alignment: Global information or spatial structure?, 2025

    Jaskirat Singh, Xingjian Leng, Zongze Wu, Liang Zheng, Richard Zhang, Eli Shechtman, and Saining Xie. What matters for representation alignment: Global information or spatial structure?, 2025. URLhttps://arxiv.org/abs/ 2512.10794

    arXiv 2025

  40. [40]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 776–780, 2017. doi: 10.1109/ICASSP.2017. 7952261

    doi:10.1109/icassp.2017 2017

  41. [41]

    BEATs: Audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. InProceedings of the 40th International Conference on Machine Learning , pages 5178–5193, 2023. URLhttps://arxiv.org/abs/2212.09058

    Pith/arXiv arXiv 2023

  42. [42]

    UCF101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild, 2012. URLhttps://arxiv.org/abs/1212.0402

    Pith/arXiv arXiv 2012

  43. [43]

    Qwen3.5: Towards native multimodal agents, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. URLhttps://qwen.ai/blog?id=qwen3.5

    2026

  44. [44]

    LibriSpeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: An ASR corpus based on public domain audio books. In2015 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5206–5210, 2015. doi: 10.1109/ICASSP.2015.7178964

    arXiv 2015

  45. [45]

    Panda-70M: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-Wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70M: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13320–13331,...

    arXiv 2024

  46. [46]

    Parker, Matthew Rice, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, Matthew Rice, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio 3,

  47. [47]

    The MUSDB18 corpus for music separation

    Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation. In International Society for Music Information Retrieval Conference, Late-Breaking/Demo Session, 2017. URLhttps://sigsep.github.io/datasets/musdb.html

    2017

  48. [48]

    Sparse in space and time: Audio-visual synchroni- sation with trainable selectors

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Sparse in space and time: Audio-visual synchroni- sation with trainable selectors. InBritish Machine Vision Conference, 2022. URLhttps://arxiv.org/abs/2210.07055

    Pith/arXiv arXiv 2022

  49. [49]

    Metaaudioboxaesthetics: Unifiedautomatic qualityassessmentforspeech,music,andsound

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, MattLe,NickZacharov,CarleighWood,AnnLee,andWei-NingHsu. Metaaudioboxaesthetics: Unifiedautomatic qualityassessmentforspeech,music,andsound. arXiv preprint arXiv:2502.05139,2025. URL https://arxiv.org/abs/ 2502.05139

    Pith/arXiv arXiv 2025

  50. [50]

    VGGSound: A large-scaleaudio-visual dataset

    HonglieChen, WeidiXie, Andrea Vedaldi, and AndrewZisserman. VGGSound: A large-scaleaudio-visual dataset. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 721–725, 2020. doi: 10.1109/ICASSP40776.2020.9053174

    arXiv 2020

  51. [51]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7482–7491, 2018. doi: 10.1109/CVPR.2018.00781. 15

    arXiv 2018

  52. [53]

    UniVerse-1: Unified audio-video generation via stitching of experts, 2025

    Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. UniVerse-1: Unified audio-video generation via stitching of experts, 2025. URLhttps://arxiv.org/abs/ 2509.06155. 14

    Pith/arXiv arXiv 2025

  53. [2024]

    doi: 10.1109/ICASSP48485.2024.10448489

    arXiv 2024

  54. [2026]

    URL https://arxiv.org/abs/2605.17991

    Pith/arXiv arXiv