{"id":"95c870a8-6d7f-4674-b620-36792bdebf83","arxiv_id":"2502.03897","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UniForm trains one diffusion transformer on a shared audio-video latent space to handle video-to-audio, audio-to-video, and text-to-audio-video generation with competitive results.","lead":"UniForm is a single diffusion transformer that can generate audio from video, video from audio, or both from text using one set of weights. It reports quality close to specialized single-task models on standard benchmarks, suggesting unified audiovisual generation is practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The A2V/V2A conditioning mechanism is described as zeroing the conditioning modality's noise input; as written this cannot carry the conditioning content, so the method's core inference step is ambiguous or incorrect.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the paper's description of how A2V and V2A condition on the input modality is ambiguous or incorrect. This is not a peripheral implementation detail; it is the mechanism by which the unified model is supposed to perform two of its three claimed tasks. The rest of the architecture, including the shared DiT, task tokens, and task-specific losses, is plausible, and the reported numbers on standard benchmarks are competitive. However, the paper does not clarify whether the conditioning modality is fed as a clean latent or as a zero tensor, and no code is released to resolve the ambiguity. If the intended implementation uses clean conditioning latents, the written procedure is inaccurate and the paper needs a correction; if it truly uses zeros, the method as described cannot work. The appropriate verdict is therefore CONDITIONAL, matching the reader's assessment, and my analysis does not move it further toward accept or reject. I also note the absence of error bars and the inconsistency between the abstract's 'close to state-of-the-art' and the tables' best scores, but these are secondary to the conditioning ambiguity because they would not invalidate the central claim even if resolved.","tokens_in":12419,"tokens_out":6405,"duration_ms":61282,"concrete_test":"Run the V2A inference path on the VGGSound test set twice: (1) with the video latent input set to the zero tensor as Section III-A states, and (2) with the clean Open-Sora video latent (no added noise) used as the conditioning input while the audio branch starts from Gaussian noise. If variant (1) fails to generate video-consistent audio (e.g., FAD close to an unconditional baseline) while variant (2) reproduces Table I's FAD=1.30 and AV-align=0.430, then the written procedure is wrong and the method depends on an unreported clean-latent conditioning step. The same check should be run for A2V on Landscape with a clean audio latent versus a zero audio latent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A defines z^a_T and z^v_T as Gaussian noise in the audio and visual modalities and Eq. (1) as the repeated denoising call. It then states that in A2V 'the audio noise input is removed by setting z^a_T = 0' and in V2A 'we set z^v_T = 0'. Taken literally, this zeros out the conditioning modality: for A2V the audio condition would be a zero tensor, and for V2A the video condition would be a zero tensor. A zero tensor carries no audio or video content, so the inference procedure as written would produce output that cannot be conditioned on the input modality. The paper never states that clean latent features of the conditioning modality (e.g., z^a_0 or z^v_0) are concatenated with the noisy target tokens, nor how such features are encoded and injected. The loss equations (10)-(11) and Figure 2 are consistent with a model that receives both modalities, but the text explicitly replaces the conditioning modality's noise input with zero, not with the clean latent. If the implementation actually feeds clean conditioning latents, the text is inaccurate; if it feeds zeros as written, the reported V2A/A2V results are not reproducible from the description. Since no code or data are released, this is the single unresolved step on which the central unified-multitask claim rests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UniForm proposes a unified Diffusion Transformer trained jointly for text-to-audio-video (T2AV), audio-to-video (A2V), and video-to-audio (V2A) generation. The model concatenates audio and video latent tokens in a shared space, adds task-specific tokens, and uses classifier-free guidance with an LLM text encoder. Experiments on VGGSound, Landscape, and AIST++ report competitive or superior scores relative to task-specific baselines on FAD, FVD, IS, and AV-align metrics.","tokens_in":12781,"tokens_out":7050,"duration_ms":65461,"significance":"If the presented design is reproducible, UniForm offers an important data point: a single parameter set can serve multiple audio-video generation tasks, with performance comparable to dedicated single-task models on several standard metrics. The use of a large-scale combined dataset and LLM-based captions is a strength. However, the core conditioning mechanism for A2V and V2A is not described precisely enough to allow replication, and the reported claims are not accompanied by uncertainty estimates.","major_comments":[{"comment":"The task-specific noise schemes state that for A2V \"the audio noise input is removed by setting z^a_T = 0\" and for V2A \"we set z^v_T = 0\". Taken literally, this zeros out the conditioning modality, so the network would receive no audio content in A2V and no video content in V2A. Since the loss functions (10)-(11) train the network on both noised modalities, the inference procedure as written cannot condition on the input modality. Please specify exactly which tensors are fed to the denoising network at each reverse step for A2V and V2A: are the conditioning latents clean (z_0) or noised (z_t)? How is the train/test distribution matched? This is a load-bearing detail for the central multi-task claim.","section":"Section III-A, Eq. (1)"},{"comment":"No error bars, confidence intervals, or significance tests are provided for any metric. Several differences are small (e.g., FAD 1.30 vs 1.34 in Table I; IS 15.43 vs 15.68), so it is impossible to judge whether the reported advantages are within run-to-run variability. Please report at least two or three seeds with means and standard deviations.","section":"Section IV, Tables I-IV"},{"comment":"The Conclusion states \"UniForm achieves state-of-the-art generation quality\", but the Abstract claims performance \"close to the state-of-the-art single-task models\", and the results in Table III show that MM-LDM achieves better FVD on both Landscape and AIST++. Please align the claims with the evidence.","section":"Section V and Abstract"}],"minor_comments":[{"comment":"\"adpot\" should be \"adopt\".","section":"Section IV.A.3"},{"comment":"The index term \"Text-to-spatial-audio\" does not reflect the paper's focus on audio-video generation; please update.","section":"Index Terms"},{"comment":"The AV-align value for VATT is listed as \"-\"; if the metric is inapplicable, state why, or report the value.","section":"Table I"},{"comment":"The reshaping of audio and video latents before concatenation is not described. Please specify how the dimensions are aligned.","section":"Section III-C.1"},{"comment":"The \"large-scale text-audio-video combined dataset\" is not quantified. Provide the number of hours or clips in the combined training set.","section":"Section IV.A.1"},{"comment":"The column header for the A2V block repeats \"FVD↓\" instead of \"KVD↓\"; please correct.","section":"Table V"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid contribution to the multimedia community if the method is clarified. The main concern is the conditioning mechanism; the authors should be asked to specify the exact input construction for A2V and V2A, including how the conditioning latents are encoded and how they are combined with the target-modal noise. Given the absence of code or data, the manuscript as written is not reproducible. I would not recommend rejection at this stage, but major revision is needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper tries to do something real—one DiT with shared weights for video-to-audio, audio-to-video, and text-to-audio-video generation. The novelty is modest but legitimate: task tokens and task-specific noise schemes on a concatenated audio-video latent. Prior work either used separate models per task or only did T2AV. The numbers are competitive, sometimes best on FAD, FD, and FVD. That is the main thing going for it.\n\nThe stress-test is right about the conditioning description. In Section III-A, for A2V they write “the audio noise input is removed by setting z^a_T = 0”, and for V2A they set z^v_T = 0. Taken literally, that zeros out the conditioning modality. A zero tensor carries no audio or video content, so the inference procedure as written cannot condition on the input. The loss equations (10)-(11) and Figure 2 look like the model does receive both modalities, so the likely truth is they feed the clean latent of the conditioning modality and only add noise to the target. But the text does not say that. This is a load-bearing implementation detail, and with no code released, the described method is not reproducible. That is the biggest soft spot.\n\nElsewhere, the experimental section lacks error bars and significance tests. Some wins are thin (IS 15.43 vs. 15.68, AV-align 0.430 vs. 0.422). The conclusion says “state-of-the-art” while the abstract says “close to state-of-the-art”, and Table III actually shows MM-LDM beating them on video metrics. That overclaim should be fixed. The ablation on text prompts and joint vs. unimodal generation is useful, and the comparison against Seeing&Hearing gives a fair picture of the unified-model trade-off.\n\nWho is this for? Researchers working on audio-video generation or unified multimodal diffusion. It is a useful baseline and the task-token idea is worth knowing. A serious referee should engage with it, mostly to force a clear description of the conditioning mechanism and variance estimates.\n\nRecommendation: send it to peer review, not desk reject, but expect heavy revision on Section III-A and the conclusion.","headline":"A genuinely unified V2A/A2V/T2AV diffusion transformer with competitive numbers, but the conditioning mechanism is described in a way that cannot work as written, and the lack of code or error bars makes the results hard to trust without a major revision.","tokens_in":13244,"tokens_out":3641,"would_cite":true,"duration_ms":37350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single diffusion transformer does all three audio-video generation tasks","keywords":["diffusion transformer","audio-video generation","multi-task generation","video-to-audio","audio-to-video","text-to-audio-video","latent diffusion","cross-modal alignment"],"falsifier":"Run audio-to-video or video-to-audio inference exactly as written, with the non-target modality's noise input set to zero, and compare generation quality against a variant that places the conditioning modality's clean latent at that position; if the zero-input version cannot produce output clearly correlated with the conditioning audio or video, the task-switching mechanism as described does not work. Inspecting the released code's inference path to see which tensor is actually passed would settle the ambiguity directly.","tokens_in":12177,"feed_emoji":"🎬","tokens_out":8462,"duration_ms":68049,"temperature":0.7,"pith_summary":"UniForm is a single diffusion transformer that generates audio and video together from text, from video, or from audio, using one set of weights. The paper claims that sharing a latent space and a denoising network across modalities lets the model capture sound-vision correlations, while task tokens and task-specific noise schemes let it switch between three generation tasks without task-specific fine-tuning. On standard benchmarks, it reports results that are close to or better than single-task specialist models on video-to-audio, audio-to-video, and text-to-audio-video, and that clearly beat the non-unified pipeline that uses separate pre-trained models per task. If this holds, one model could replace three separate generative systems while improving cross-modal alignment.","feed_headline":"One transformer does all three audio-video generation tasks","feed_subtitle":"UniForm: one weight set matches specialist models for video-to-audio, audio-to-video, and joint text-to-AV.","key_machinery":"The central object is a unified multimodal latent space formed by concatenating audio latents from a pre-trained audio VAE and video latents from a pre-trained video VAE along the token dimension, processed by STDiT3 blocks that apply spatial and temporal attention plus cross-attention to text. Task tokens specify which generation task is active, while task-specific noise schemes set the non-target modality's initial noise to zero and mask-based losses extract the audio or video prediction from the joint output. This machinery lets one denoising network generate both modalities and align them during joint diffusion.","core_discovery":"The central claim is that a single DiT-based denoising network operating on concatenated audio and video latents can perform video-to-audio, audio-to-video, and text-to-audio-video generation with one set of parameters. The paper introduces task tokens to tell the model which task is active and task-specific noise schemes: for audio-to-video the audio noise input is set to zero, and for video-to-audio the video noise input is set to zero, leaving the other modality's latent to serve as the condition. Text conditioning is optional in all tasks, with classifier-free guidance dropping the text 50% of the time. Experiments report the strongest FAD (1.30) and FD (6.21) on video-to-audio, the strongest FVD (219) and IS (4.61) on audio-to-video, and the strongest FAD (2.41 on Landscape, 1.27 on AIST++) on text-to-audio-video, close to or better than single-task baselines and consistently better than the non-unified Seeing&Hearing baseline.","pith_inferences":["A direct testable extension would be to swap the zero-tensor noise scheme for explicit cross-modal conditioning, feeding clean latents of the conditioning modality, and measure whether audio-to-video and video-to-audio quality improve further.","If the single-network sharing claim holds, the same architecture could scale to extra modalities or conditions, such as camera motion or depth signals, by adding tokens and noise schedules rather than new generators.","The reported alignment gains from joint generation over unimodal generation suggest the shared denoiser itself, not the text captions, drives audio-video synchronization; ablating text entirely on both versions would test this.","The automatically generated caption corpus is a reusable resource whose effect could be isolated by training the same architecture on original dataset labels versus the LLM captions and comparing diversity metrics."],"forward_implications":["A single UniForm model can replace three separate task-specific generators for video-to-audio, audio-to-video, and text-to-audio-video without fine-tuning per task.","Joint training in a shared latent space improves audio-video alignment compared with independently generating audio and video using the same architecture.","Optional text prompts improve FAD, FD, FVD, IS, and KL on both video-to-audio and audio-to-video, with AV-align roughly unchanged.","Training on a large-scale text-audio-video dataset with LLM-generated captions supports diverse scenes such as game footage and speech-synchronized portraits.","UniForm's reported benchmark numbers are competitive with or better than single-task specialist models and superior to the non-unified multi-task baseline."],"supporting_citations":[{"why":"Supplies the diffusion transformer backbone that UniForm adapts to the multimodal setting.","marker":"[14]"},{"why":"Supplies the latent diffusion formulation used to shift denoising into a compressed latent space.","marker":"[3]"},{"why":"Supplies the video VAE encoder/decoder and the STDiT3 block design used for spatial-temporal modeling.","marker":"[8]"},{"why":"Supplies the pre-trained audio VAE used to encode and reconstruct mel-spectrogram latents.","marker":"[20]"},{"why":"The non-unified multi-task baseline, using separate pre-trained models per task, that UniForm compares against across all three tasks.","marker":"[17]"},{"why":"Provides the classifier-free guidance strategy used for text-conditioned generation.","marker":"[33]"},{"why":"Provides the T5 text encoder used to embed prompt text into the shared denoising network.","marker":"[35]"},{"why":"Provides the instruction-tuned FLAN-T5 variant used as the text encoder for the model.","marker":"[1]"},{"why":"Supplies the HiFi-GAN vocoder that converts generated mel-spectrograms into audio waveforms.","marker":"[36]"}],"fun_headline_variants":["One transformer unifies all three audio-video generation tasks","UniForm: a single model matches specialists in audio-video generation","Multi-task diffusion transformer: audio and video in one shared space","From video to audio, audio to video: one transformer does it all","Single DiT for video-to-audio, audio-to-video, and text-to-AV generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that, for audio-to-video and video-to-audio, feeding a zero tensor as the non-target modality's initial noise lets the denoising network condition on the other modality; a zero tensor carries no content, and the paper never clarifies whether the conditioning modality is actually supplied as clean latent tokens instead of zeros.","fun_headline_variants_meta":{"raw":{"variants":["One transformer unifies all three audio-video generation tasks","UniForm: a single model matches specialists in audio-video generation","Multi-task diffusion transformer: audio and video in one shared space","From video to audio, audio to video: one transformer does it all","Single DiT for video-to-audio, audio-to-video, and text-to-AV generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1807,"prompt_tokens":964,"completion_tokens":843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":746}},"tokens_in":580,"tokens_out":843,"duration_ms":7866,"temperature":1.0,"reasoning_tokens":746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:18:33.158055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run audio-to-video or video-to-audio inference exactly as written, with the non-target modality's noise input set to zero, and compare generation quality against a variant that places the conditioning modality's clean latent at that position; if the zero-input version cannot produce output clearly correlated with the conditioning audio or video, the task-switching mechanism as described does not work. Inspecting the released code's inference path to see which tensor is actually passed would settle the ambiguity directly.","supporting_citations":[{"cited_title":"Scalable diffusion models with trans- formers,","cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion transformer backbone that UniForm adapts to the multimodal setting."},{"cited_title":"High-resolution image synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion formulation used to shift denoising into a compressed latent space."},{"cited_title":"Audioldm: Text-to-audio generation with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained audio VAE used to encode and reconstruct mel-spectrogram latents."},{"cited_title":"Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,","cited_arxiv_id":null,"evidence_quote":"The non-unified multi-task baseline, using separate pre-trained models per task, that UniForm compares against across all three tasks."},{"cited_title":"Classifier-free diffusion guidance,","cited_arxiv_id":null,"evidence_quote":"Provides the classifier-free guidance strategy used for text-conditioned generation."},{"cited_title":"Scaling instruction-finetuned language models,","cited_arxiv_id":null,"evidence_quote":"Provides the instruction-tuned FLAN-T5 variant used as the text encoder for the model."},{"cited_title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the HiFi-GAN vocoder that converts generated mel-spectrograms into audio waveforms."}],"review_version":1}