Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single diffusion transformer does all three audio-video generation tasks

desk verdict A genuinely unified V2A/A2V/T2AV diffusion transformer with competitive numbers, but the conditioning mechanism is described in a way that cannot work as written, and the lack of code or error bars makes the results hard to trust without a major revision. read the letter →

arxiv 2502.03897 v5 pith:H4QPS3XC submitted 2025-02-06 cs.MM cs.AIcs.CVcs.SDeess.AS

Lei Zhao , Linfeng Feng , Dongxu Ge , Rujin Chen , Fangqiu Yi , Chi Zhang , Xiao-Lei Zhang , Xuelong Li This is my paper · ORCID
classification cs.MMcs.AIcs.CVcs.SDeess.AS
keywords diffusiontransformeraudio-videogenerationmulti-taskvideo-to-audioaudio-to-videotext-to-audio-videolatentcross-modalalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniForm is a single diffusion transformer that generates audio and video together from text, from video, or from audio, using one set of weights. The paper claims that sharing a latent space and a denoising network across modalities lets the model capture sound-vision correlations, while task tokens and task-specific noise schemes let it switch between three generation tasks without task-specific fine-tuning. On standard benchmarks, it reports results that are close to or better than single-task specialist models on video-to-audio, audio-to-video, and text-to-audio-video, and that clearly beat the non-unified pipeline that uses separate pre-trained models per task. If this holds, one model could replace three separate generative systems while improving cross-modal alignment.

What carries the argument

The central object is a unified multimodal latent space formed by concatenating audio latents from a pre-trained audio VAE and video latents from a pre-trained video VAE along the token dimension, processed by STDiT3 blocks that apply spatial and temporal attention plus cross-attention to text. Task tokens specify which generation task is active, while task-specific noise schemes set the non-target modality's initial noise to zero and mask-based losses extract the audio or video prediction from the joint output. This machinery lets one denoising network generate both modalities and align them during joint diffusion.

What would settle it

Run audio-to-video or video-to-audio inference exactly as written, with the non-target modality's noise input set to zero, and compare generation quality against a variant that places the conditioning modality's clean latent at that position; if the zero-input version cannot produce output clearly correlated with the conditioning audio or video, the task-switching mechanism as described does not work. Inspecting the released code's inference path to see which tensor is actually passed would settle the ambiguity directly.

Watch

Extended reading notes

Core claim

The central claim is that a single DiT-based denoising network operating on concatenated audio and video latents can perform video-to-audio, audio-to-video, and text-to-audio-video generation with one set of parameters. The paper introduces task tokens to tell the model which task is active and task-specific noise schemes: for audio-to-video the audio noise input is set to zero, and for video-to-audio the video noise input is set to zero, leaving the other modality's latent to serve as the condition. Text conditioning is optional in all tasks, with classifier-free guidance dropping the text 50% of the time. Experiments report the strongest FAD (1.30) and FD (6.21) on video-to-audio, the strongest FVD (219) and IS (4.61) on audio-to-video, and the strongest FAD (2.41 on Landscape, 1.27 on AIST++) on text-to-audio-video, close to or better than single-task baselines and consistently better than the non-unified Seeing&Hearing baseline.

Load-bearing premise

The method assumes that, for audio-to-video and video-to-audio, feeding a zero tensor as the non-target modality's initial noise lets the denoising network condition on the other modality; a zero tensor carries no content, and the paper never clarifies whether the conditioning modality is actually supplied as clean latent tokens instead of zeros.

Editorial extensions

If this is right

  • A single UniForm model can replace three separate task-specific generators for video-to-audio, audio-to-video, and text-to-audio-video without fine-tuning per task.
  • Joint training in a shared latent space improves audio-video alignment compared with independently generating audio and video using the same architecture.
  • Optional text prompts improve FAD, FD, FVD, IS, and KL on both video-to-audio and audio-to-video, with AV-align roughly unchanged.
  • Training on a large-scale text-audio-video dataset with LLM-generated captions supports diverse scenes such as game footage and speech-synchronized portraits.
  • UniForm's reported benchmark numbers are competitive with or better than single-task specialist models and superior to the non-unified multi-task baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be to swap the zero-tensor noise scheme for explicit cross-modal conditioning, feeding clean latents of the conditioning modality, and measure whether audio-to-video and video-to-audio quality improve further.
  • If the single-network sharing claim holds, the same architecture could scale to extra modalities or conditions, such as camera motion or depth signals, by adding tokens and noise schedules rather than new generators.
  • The reported alignment gains from joint generation over unimodal generation suggest the shared denoiser itself, not the text captions, drives audio-video synchronization; ablating text entirely on both versions would test this.
  • The automatically generated caption corpus is a reusable resource whose effect could be isolated by training the same architecture on original dataset labels versus the LLM captions and comparing diversity metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. UniForm proposes a unified Diffusion Transformer trained jointly for text-to-audio-video (T2AV), audio-to-video (A2V), and video-to-audio (V2A) generation. The model concatenates audio and video latent tokens in a shared space, adds task-specific tokens, and uses classifier-free guidance with an LLM text encoder. Experiments on VGGSound, Landscape, and AIST++ report competitive or superior scores relative to task-specific baselines on FAD, FVD, IS, and AV-align metrics.

Significance. If the presented design is reproducible, UniForm offers an important data point: a single parameter set can serve multiple audio-video generation tasks, with performance comparable to dedicated single-task models on several standard metrics. The use of a large-scale combined dataset and LLM-based captions is a strength. However, the core conditioning mechanism for A2V and V2A is not described precisely enough to allow replication, and the reported claims are not accompanied by uncertainty estimates.

major comments (3)
  1. [Section III-A, Eq. (1)] The task-specific noise schemes state that for A2V "the audio noise input is removed by setting z^a_T = 0" and for V2A "we set z^v_T = 0". Taken literally, this zeros out the conditioning modality, so the network would receive no audio content in A2V and no video content in V2A. Since the loss functions (10)-(11) train the network on both noised modalities, the inference procedure as written cannot condition on the input modality. Please specify exactly which tensors are fed to the denoising network at each reverse step for A2V and V2A: are the conditioning latents clean (z_0) or noised (z_t)? How is the train/test distribution matched? This is a load-bearing detail for the central multi-task claim.
  2. [Section IV, Tables I-IV] No error bars, confidence intervals, or significance tests are provided for any metric. Several differences are small (e.g., FAD 1.30 vs 1.34 in Table I; IS 15.43 vs 15.68), so it is impossible to judge whether the reported advantages are within run-to-run variability. Please report at least two or three seeds with means and standard deviations.
  3. [Section V and Abstract] The Conclusion states "UniForm achieves state-of-the-art generation quality", but the Abstract claims performance "close to the state-of-the-art single-task models", and the results in Table III show that MM-LDM achieves better FVD on both Landscape and AIST++. Please align the claims with the evidence.
minor comments (6)
  1. [Section IV.A.3] "adpot" should be "adopt".
  2. [Index Terms] The index term "Text-to-spatial-audio" does not reflect the paper's focus on audio-video generation; please update.
  3. [Table I] The AV-align value for VATT is listed as "-"; if the metric is inapplicable, state why, or report the value.
  4. [Section III-C.1] The reshaping of audio and video latents before concatenation is not described. Please specify how the dimensions are aligned.
  5. [Section IV.A.1] The "large-scale text-audio-video combined dataset" is not quantified. Provide the number of hours or clips in the combined training set.
  6. [Table V] The column header for the A2V block repeats "FVD↓" instead of "KVD↓"; please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found; UniForm's reported results rest on external benchmarks and standard diffusion objectives, with only a non-circular reproducibility ambiguity in the A2V/V2A zero-input conditioning description.

full rationale

The core derivation chain is not circular. Eq. (5) and Eqs. (10)-(12) are standard diffusion noise-prediction losses: the network predicts the noise added to audio/video latents, and the reported FAD/FD/FVD/IS/AV-align scores are computed on held-out test sets (VGGSound, Landscape, AIST++) against external single-task baselines. None of the reported quantities is produced by fitting a parameter to the same quantity and then relabeling it as a prediction; there is no fitted constant, no imported uniqueness theorem, and no ansatz smuggled in through a self-citation. The pretrained components (Open-Sora video VAE, AudioLDM audio VAE, FLAN-T5, HiFi-GAN, PixArt-delta weights) are external, frozen scaffolding rather than a self-referential loop. The only passage that deserves scrutiny is Section III-A, where the paper says 'the audio noise input is removed by setting z^a_T = 0' for A2V and 'we set z^v_T = 0' for V2A. Taken literally, a zero tensor cannot carry the conditioning audio or video content, so the inference procedure as written is either under-specified or incorrect. However, this is a reproducibility/correctness defect, not a circularity: it does not make any metric or generation result equal to its own input by construction. The measured comparison with external baselines would remain meaningful whether the implementation feeds zeros or clean conditioning latents. Self-citation is essentially absent and never load-bearing; the few references to work by co-authors (e.g., [4], [6]) are not used to justify the architecture or to forbid alternatives. The training-split choice follows [15], but that is standard evaluation practice and does not inject the target result. Accordingly, the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The model relies on pretrained components (VAEs, FLAN-T5, pLLaVA, PixArt-delta DiT) and standard diffusion assumptions. The only new architectural elements are task tokens and the concatenated latent space, which are domain assumptions rather than independently evidenced entities. No new physical or mathematical objects are postulated.

free parameters (3)
  • Classifier-free guidance scale = 5
    Set during inference for all tasks; affects the tradeoff between sample quality and diversity, not derived from first principles.
  • Number of inference steps = 30
    Uniformly set for all tasks; standard DDIM-style choice, not justified by analysis.
  • Text conditioning dropout probability = 0.5
    Used for classifier-free guidance training; chosen as a common default, not tuned per task.
assumptions (4)
  • domain assumption The frozen pretrained VAEs (Open-Sora for video, AudioLDM for audio) provide a sufficiently expressive and compatible latent space for joint diffusion.
    The method concatenates latents from two different VAEs and relies on their compatibility without any alignment or joint training of the encoders.
  • ad hoc to paper Concatenating audio and video latent tokens along the last dimension forms a valid unified representation for the DiT.
    This is the core design choice; the paper provides no theoretical or empirical analysis showing that concatenation preserves cross-modal relationships better than alternatives.
  • domain assumption Task tokens provide sufficient information for the network to switch between generation tasks.
    The paper assumes that a learned task embedding can disambiguate the three tasks during joint training; no ablation shows this is necessary or sufficient.
  • domain assumption The combination of VGGSound, Landscape, AIST++, AudioSet-balance, and AudioSet-Strong is a beneficial training corpus for multi-task learning.
    The paper asserts the benefit of 'large-scale data' but does not isolate the effect of each dataset or test whether dataset imbalance harms any single task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation." pith.science (2026). https://pith.science/paper/H4QPS3XC

@misc{pith2026250203897,
  author       = {Pith},
  title        = {Pith review of: UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H4QPS3XC}},
  note         = {Machine review of arXiv:2502.03897}
}
read the original abstract

With the rise of diffusion models, audio-video generation has been revolutionized. However, most existing methods rely on separate modules for each modality, with limited exploration of unified generative architectures. In addition, many are confined to a single task and small-scale datasets. To overcome these limitations, we introduce UniForm, a unified multi-task diffusion transformer that generates both audio and visual modalities in a shared latent space. By using a unified denoising network, UniForm captures the inherent correlations between sound and vision. Additionally, we propose task-specific noise schemes and task tokens, enabling the model to support multiple tasks with a single set of parameters, including video-to-audio, audio-to-video and text-to-audio-video generation. Furthermore, by leveraging large language models and a large-scale text-audio-video combined dataset, UniForm achieves greater generative diversity than prior approaches. Experiments show that UniForm achieves performance close to the state-of-the-art single-task models across three generation tasks, with generated content that is not only highly aligned with real-world data distributions but also enables more diverse and fine-grained generation.

Figures

Figures reproduced from arXiv: 2502.03897 by the authors.

Figure 1
Figure 1. Illustration of multimodal-conditioned audio-video gen [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed UniForm. Vision tokens and audio tokens are integrated and processed within a unified [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Compared with FoleyCrafter in V2A generation on the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Generated samples in the A2V task on the Landscape dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Generated samples in the T2AV task on the Landscape dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Generated samples in the T2AV task on the AIST++ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Generated two challenging samples on the VGGSound dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vorch-Omni: Multi-Task Orchestration of Sight and Sound

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A single flow-matching diffusion transformer with explicit token roles and position types performs more than 30 audio-visual generation, editing, and reference-conditioned tasks without task-specific architectures.

  2. Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Separating the text condition into a video caption and a visually grounded audio caption, and fusing the diffusion towers with dual cross-attention, gives the reported-best text-to-sounding-video quality and synchroni...

  3. UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A unified audio-video generator built by stitching pre-trained video and music diffusion models, trained on 7,600 hours of data, with a new evaluation benchmark.

  4. JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1

    cs.CV 2025-07 reject novelty 4.0 of 10

    A paper announcing a large-scale whole-body talking avatar benchmark and evaluation protocol, but with insufficient details to verify the dataset or the joint audio-video evaluation.

Reference graph

Works this paper leans on

43 extracted references · 29 canonical work pages · cited by 4 Pith papers

  1. [1]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma et al. , “Scaling instruction-finetuned language models,” Journal of Machine Learning Research, vol. 25, no. 70, pp. 1–53, 2024

    work page 2024

  2. [2]

    Enhanced visual instruction tuning with synthesized image-dialogue data,

    Y . Li, C. Zhang, G. Yu, W. Yang, Z. Wang, B. Fu, G. Lin, C. Shen, L. Chen, and Y . Wei, “Enhanced visual instruction tuning with synthesized image-dialogue data,” in Findings of the Association for Computational Linguistics ACL 2024 , 2024, pp. 14 512–14 531

    work page 2024

  3. [3]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684–10 695

    work page 2022

  4. [4]

    Label-guided generative adversarial network for realistic image synthesis,

    J. Zhu, L. Gao, J. Song, Y .-F. Li, F. Zheng, X. Li, and H. T. Shen, “Label-guided generative adversarial network for realistic image synthesis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3311–3328, 2023

    work page 2023

  5. [5]

    Audioldm 2: Learn- ing holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learn- ing holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 32, pp. 2871–2883, 2024

    2024

  6. [6]

    Continuous emotion-based image-to-music generation,

    Y . Wang, M. Chen, and X. Li, “Continuous emotion-based image-to-music generation,” IEEE Transactions on Multimedia, vol. 26, pp. 5670–5679, 2024

    work page 2024

  7. [7]

    Moviellm: Enhancing long video understanding with ai-generated movies,

    Z. Song, C. Wang, J. Sheng, C. Zhang, G. Yu, J. Fan, and T. Chen, “Moviellm: Enhancing long video understanding with ai-generated movies,” arXiv preprint arXiv:2403.01422 , 2024. 9

    arXiv 2024

  8. [8]

    Open-sora: Democratizing efficient video production for all,

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y . Zhou, T. Li, and Y . You, “Open-sora: Democratizing efficient video production for all,” arXiv preprint arXiv:2412.20404 , 2024

    arXiv 2024

Show all 43 references
  1. [9]

    Make-a-video: Text-to-video generation without text-video data,

    U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y . Taigman, “Make-a-video: Text-to-video generation without text-video data,” in The Eleventh International Conference on Learning Representations , 2023. [O...

    2023

  2. [10]

    Align your latents: High-resolution video synthesis with latent diffusion models,

    A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 2...

    2023

  3. [11]

    Lavie: High-quality video generation with cascaded latent diffusion models,

    Y . Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y . Wang, C. Yang, Y . He, J. Yu, P. Yanget al., “Lavie: High-quality video generation with cascaded latent diffusion models,” International Journal of Computer Vision , pp. 1–20, 2024

    2024

  4. [12]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,

    L. Ruan, Y . Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo, “Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 219–10 228

    2023

  5. [13]

    Mm-ldm: Multi-modal latent diffusion model for sounding video generation,

    M. Sun, W. Wang, Y . Qiao, J. Sun, Z. Qin, L. Guo, X. Zhu, and J. Liu, “Mm-ldm: Multi-modal latent diffusion model for sounding video generation,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 10 853– 10 861

    2024

  6. [14]

    Scalable diffusion models with trans- formers,

    W. Peebles and S. Xie, “Scalable diffusion models with trans- formers,” in Proceedings of the IEEE/CVF International Con- ference on Computer Vision , 2023, pp. 4195–4205

    2023

  7. [15]

    Mmdisco: Multi-modal discriminator-guided cooperative diffusion for joint audio and video generation,

    A. Hayakawa, M. Ishii, T. Shibuya, and Y . Mitsufuji, “Mmdisco: Multi-modal discriminator-guided cooperative diffusion for joint audio and video generation,” in The Thirteenth Interna- tional Conference on Learning Representations , 2025

    2025

  8. [16]

    Av-dit: Efficient audio-visual diffusion transformer for joint audio and video generation,

    K. Wang, S. Deng, J. Shi, D. Hatzinakos, and Y . Tian, “Av-dit: Efficient audio-visual diffusion transformer for joint audio and video generation,” arXiv preprint arXiv:2406.07686 , 2024

    2024 arXiv

  9. [17]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,

    Y . Xing, Y . He, Z. Tian, X. Wang, and Q. Chen, “Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7151– 7161

    2024

  10. [18]

    Foley sound synthesis at the dcase 2023 challenge,

    K. Choi, J. Im, L. Heller, B. McFee, K. Imoto, Y . Okamoto, M. Lagrange, and S. Takamichi, “Foley sound synthesis at the dcase 2023 challenge,” In arXiv e-prints: 2304.12521 , 2023

    2023 arXiv

  11. [19]

    Conditional sound generation using neural discrete time-frequency representation learning,

    X. Liu, T. Iqbal, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Conditional sound generation using neural discrete time-frequency representation learning,” in 2021 IEEE 31st International Workshop on Machine Learning for Signal Pro- cessing (MLSP). IEEE, 2021, pp. 1–6

    2021

  12. [20]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” in Proceedings of the 40th Interna- tional Conference on Machine Learning , vol. 202, 2023, pp. 21 450–21 474

    2023

  13. [21]

    Taming visually guided sound gener- ation,

    V . Iashin and E. Rahtu, “Taming visually guided sound gener- ation,” arXiv preprint arXiv:2110.08791 , 2021

    2021 arXiv

  14. [22]

    Con- ditional generation of audio from video via foley analogies,

    Y . Du, Z. Chen, J. Salamon, B. Russell, and A. Owens, “Con- ditional generation of audio from video via foley analogies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2426–2436

    2023

  15. [23]

    Diff-foley: Synchro- nized video-to-audio synthesis with latent diffusion models,

    S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchro- nized video-to-audio synthesis with latent diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  16. [24]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,

    Y . Zhang, Y . Gu, Y . Zeng, Z. Xing, Y . Wang, Z. Wu, and K. Chen, “Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,” arXiv preprint arXiv:2407.01494 , 2024

    2024 arXiv

  17. [25]

    Temporally aligned audio for video with autoregression,

    I. Viertola, V . Iashin, and E. Rahtu, “Temporally aligned audio for video with autoregression,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  18. [26]

    Tell what you hear from what you see–video to audio generation through text,

    X. Liu, K. Su, and E. Shlizerman, “Tell what you hear from what you see–video to audio generation through text,” arXiv preprint arXiv:2411.05679, 2024

    2024 arXiv

  19. [27]

    Frieren: Efficient video-to-audio generation network with rectified flow matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation network with rectified flow matching,” In NeurIPS, 2024

    2024

  20. [28]

    Sound-guided semantic video generation,

    S. H. Lee, G. Oh, W. Byeon, C. Kim, W. J. Ryoo, S. H. Yoon, H. Cho, J. Bae, J. Kim, and S. Kim, “Sound-guided semantic video generation,” in European Conference on Computer Vision. Springer, 2022, pp. 34–50

    2022

  21. [29]

    The power of sound (tpos): Audio reactive video gen- eration with stable diffusion,

    Y . Jeong, W. Ryoo, S. Lee, D. Seo, W. Byeon, S. Kim, and J. Kim, “The power of sound (tpos): Audio reactive video gen- eration with stable diffusion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7822– 7832

    2023

  22. [30]

    Diverse and aligned audio-to-video generation via text-to-video model adaptation,

    G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y . Adi, “Diverse and aligned audio-to-video generation via text-to-video model adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 6639–6647

    2024

  23. [31]

    Hi- erarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hi- erarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125 , vol. 1, no. 2, p. 3, 2022

    2022 arXiv

  24. [32]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020

    2020

  25. [33]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021

    2021

  26. [34]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

    2021

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  28. [36]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Ad- vances in neural information processing systems , vol. 33, pp. 17 022–17 033, 2020

    2020

  29. [37]

    Vggsound: A large-scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2020, pp. 721–725

    2020

  30. [38]

    Ai choreog- rapher: Music conditioned 3d dance generation with aist++,

    R. Li, S. Yang, D. A. Ross, and A. Kanazawa, “Ai choreog- rapher: Music conditioned 3d dance generation with aist++,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 13 401–13 412

    2021

  31. [39]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2017, ...

    2017

  32. [40]

    The benefit of temporally-strong labels in audio event classification,

    S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The benefit of temporally-strong labels in audio event classification,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2021, pp. 366–370

    2021

  33. [41]

    Aist dance video database: Multi-genre, multi-dancer, and multi- camera database for dance information processing

    S. Tsuchida, S. Fukayama, M. Hamasaki, and M. Goto, “Aist dance video database: Multi-genre, multi-dancer, and multi- camera database for dance information processing.” in ISMIR, vol. 1, no. 5, 2019, p. 6

    2019

  34. [42]

    Pllava: 10 Parameter-free llava extension from images to videos for video dense captioning,

    L. Xu, Y . Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng, “Pllava: 10 Parameter-free llava extension from images to videos for video dense captioning,” arXiv preprint arXiv:2404.16994 , 2024

    2024 arXiv

  35. [43]

    Pixart- δ: Fast and controllable image generation with latent consistency models,

    J. Chen, Y . Wu, S. Luo, E. Xie, S. Paul, P. Luo, H. Zhao, and Z. Li, “Pixart- δ: Fast and controllable image generation with latent consistency models,” arXiv preprint arXiv:2401.05252 , 2024

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.