Pith. sign in

REVIEW 4 major objections 7 minor 17 references

Flowley claims that a single-stage, alignment-module-free flow-matching model can generate state-of-the-art video-synchronized audio by using a progressive soft mask in cross-attention, and that richer captions extend the edge to zero-shot

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:17 UTC pith:7H47QJAJ

load-bearing objection Single-stage 169M Flowley with the PSCA mask reports strong VGGSound numbers and SoundCap transfers to MMAudio, but no error bars and a hard-coded alignment window make the SOTA claim conditional. the 4 major comments →

arxiv 2607.06405 v2 pith:7H47QJAJ submitted 2026-07-07 cs.MM cs.SD

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Thanh V. T. Tran , Ngoc-Son Nguyen , Luong Tran , Long-Khanh Pham , Paarth Neekhara , Shehzeen Hussain , Van Nguyen This is my paper
classification cs.MM cs.SD
keywords video-to-audio generationflow matchingcross-attention maskingtemporal alignmentsound-aware captioninglatent space generationmultimodal conditioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Flowley is a single-stage, end-to-end video-to-audio system built on flow matching. The paper's central claim is that precise audiovisual synchronization can be learned inside the attention mechanism itself, so pretrained alignment networks, onset detectors, and multi-stage training become unnecessary. Synchronization is carried by Progressive Soft-masked Cross-Attention (PSCA), which restricts each audio token to a narrow, depth-shrinking window of video frames around the frame predicted by a fixed audio/video sampling ratio. On the VGGSound benchmark, the paper reports that a 169M-parameter Flowley outperforms all compared methods on distribution matching, audio quality, and semantic alignment while remaining competitive on a synchronization metric. The companion SoundCap pipeline generates detailed sound-oriented captions from silent video; with these captions, Flowley's zero-shot audio quality is reported to exceed that of a closed-source model roughly 77 times larger trained on far more data.

Core claim

The paper argues that temporal alignment can be embedded in the flow model rather than added as a module. PSCA modifies visual cross-attention by adding a log-mask to the attention logits: mask value is 1 for the video frame aligned to audio token i under j_c(i)=floor(r_v i/r_a), decays with a cosine over a hand-set fade zone beyond a hard window, and the whole fade zone is scaled by beta that decreases with block depth. This makes early layers attend to a broad context and later layers converge to a hard local window, all at zero extra parameter or computational cost. The claimed consequence is state-of-the-art results on VGGSound across multiple metrics and competitive or better human pref

What carries the argument

The central mechanism is the Progressive Soft-masked Cross-Attention (PSCA) layer. Given audio tokens at rate r_a and video frames at rate r_v, each audio position i is assigned an aligned frame j_c(i)=min(floor(r_v i/r_a), L_vis-1); attention weight to frame j is masked by M=1 for |j-j_c|≤ω, by a cosine decay (1+cos(π m/δ))/2 over the next δ frames, and 0 beyond, with the decay region scaled by β_l=1-l/(N2-1). Since log(M+ε) is added to QK^T, the mask differentiably enforces locality and grows stricter with depth. The mechanism embeds a monotonic frame-rate-linear alignment prior directly in the latent flow model, replacing dedicated alignment networks at no parameter cost; a learnable scal

Load-bearing premise

The load-bearing premise is that the sound for each moment comes from the video frame that a fixed sampling-rate ratio predicts, within a narrow hand-set window; the paper offers no measurement of how often real sounds break that assumption.

What would settle it

Annotate a random sample of VGGSound clips with the true onset video frame of each visible sound-producing event and count how many fall outside PSCA's ω+δ window under the paper's 8 FPS mapping; if a substantial fraction do, the model's own alignment prior should suppress the correct video cues and its alignment accuracy should drop on exactly those clips. A cleaner test: evaluate Flowley on a held-out set of videos with systematically delayed or off-screen sounds and compare alignment accuracy to clips whose sounds are frame-aligned; the paper's claim predicts the two should behave similarly

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, synchronized video-to-audio can be trained in one stage with a vanilla flow-matching objective; dedicated audio-visual alignment encoders and contrastive pretraining stages are dispensable.
  • PSCA adds zero parameters and zero added attention cost, so any transformer-based audio generator can adopt the same mask to enforce hard temporal locality during generation.
  • Rich sound-oriented conditioning, produced without access to ground-truth audio at inference, improves distribution matching and semantic fidelity, and the benefit transfers to at least one other architecture.
  • Small models trained on a few hundred hours of data can match or exceed much larger closed systems on zero-shot audio quality, though semantic alignment on out-of-distribution content remains lower.
  • The paper's own limitation section concedes that speech generation remains unintelligible, background music quality is limited, and out-of-distribution generalization is restricted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because PSCA's alignment map is fixed by frame-rate ratios rather than learned, the method would likely struggle with genuinely off-screen or delayed sound sources; a testable extension is to make the mask center or width a learned function of audio-visual similarity.
  • The success of a hand-set cosine mask suggests attention-mask priors could replace alignment pretraining in other monotonic multimodal tasks, such as event-sound retrieval or lip-sync, where the correspondence is approximately linear in time.
  • SoundCap's gains suggest recaptioning noisy in-the-wild benchmarks may be a cheap, dataset-independent performance lever; one could test whether downstream KAD/FAD improves monotonically with caption specificity across multiple caption sources.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. Flowley is a flow-matching video-to-audio model trained end-to-end in a single stage. It uses multi-stream MM-DiT-style blocks followed by single-stream DiT blocks; in each single-stream block, text is attended by ordinary cross-attention and video by Progressive Soft-masked Cross-Attention (PSCA), which adds a multiplicative soft mask to the attention logits. The mask is centered at a hard-coded linear mapping from audio-token index to video-frame index, with a hard window ω and a cosine fade zone δ that shrinks with layer depth. SoundCap is a companion pipeline that uses the AV-LLM video-SALMONN to write detailed sound-aware captions and then fine-tunes Qwen2.5-VL to produce captions from silent video. On VGGSound, Flowley reports the best KAD/FAD/KL/IS/IB-Score/LB-Score among the compared baselines and competitive Align Acc; with SoundCap it reports a higher Inception Score than Movie Gen Audio on the Movie Gen Audio Bench.

Significance. If the reported numbers are reliable, the contribution is practically meaningful: a compact 169M-parameter single-stage model without external audio-visual alignment modules reaches top results on standard VGGSound metrics, and the SoundCap recaptioning pipeline transfers to a second architecture (MMAudio). The paper includes standard ablations that are largely consistent with the claims: PSCA improves Align Acc (Table 3, rows 4 vs 6), the layer-progressive β helps (Table 4), and the noise-aware prompt design is clearly beneficial (Table 5). The tangent-based sampling schedule is derived cleanly in the supplement, and the human-preference study is a useful complement to automatic metrics. The main weaknesses are that the central PSCA alignment prior is under-validated empirically and some headline claims are stated more strongly than the evidence supports; these are fixable with additional analysis and softened wording.

major comments (4)
  1. [§3.3, Eq. (5); §4.1] PSCA hard-codes temporal alignment as jc(i)=floor(rv·i/ra) with defaults ω=0 and δ=4, i.e., each audio token can attend only to video frames within roughly ±0.5 s of the linearly predicted center at 8 fps. The paper never reports the distribution of true audio-visual offsets in VGGSound, and Table 2 only varies ω and δ; it never varies the mapping itself. The experiment thus cannot distinguish “the mask helps” from “the mask helps only when the linear prior is correct.” Since PSCA is the mechanism that is claimed to eliminate external alignment modules, the authors should (a) measure how often ground-truth sound onsets fall outside the ω+δ window, and (b) ablate the mapping itself (e.g., global shift, learned per-layer effective offset, asymmetric windows, or an unmasked visual cross-attention baseline with otherwise identical architecture). Without this, the central synchronization clai
  2. [Abstract; §4.2, Table 6] The abstract states that with SoundCap, Flowley “exceeds the performance of the strongest existing close-sourced methods in terms of audio quality.” In Table 6 this rests on a single metric, IS (8.18 vs 8.01); on the same table, IB-Score (25.78 vs 35.86) and Align Acc (61.07 vs 64.33) are substantially lower. Because IS is a classifier-based score rather than a direct audio-quality measure, and no confidence intervals or per-sample variance are reported, the headline is stronger than the evidence. Please either restrict the claim to “higher Inception Score” on this benchmark, add significance tests/error bars, or provide human evaluation on Movie Gen Audio Bench.
  3. [§3.5; Supplementary §4] SoundCap is motivated by “capturing both on- and off-screen audio events,” but at inference the fine-tuned VLM sees only silent video. A truly off-screen event is not visually recoverable, and PSCA then prevents the audio latent from attending to any video frame farther than ω+δ from the linear center. The paper does not explain how off-screen events are expected to reach the generator (presumably only through the text path) and provides no analysis of whether the VLM captions actually recover such events. The Limitations section (Supp. §4) acknowledges speech, music, and OOD generalization but not this failure mode. Please define the intended mechanism for off-screen audio and quantify how often SoundCap captions contain events not inferable from the visible frames.
  4. [Tables 1–4] All comparative results are single-run point estimates without confidence intervals or significance tests. Several differences that carry the ablations are small, e.g., Align Acc 89.37 vs 88.62 in Table 4 and KAD 0.42 vs 0.40 in Table 3, row 5 vs row 6. Given that the central claims are comparative, the authors should report variance over seeds or a significance test (e.g., bootstrap over the test set for CAVP-based Align Acc), especially for the KAD and Align Acc differences used to justify PSCA and β.
minor comments (7)
  1. [§4.3, Table 2] The sentence “Increasing ω and δ generally leads to lower Align Acc” is not supported by the table: the default (ω=0, δ=4) is the best row (89.37), while δ=0 and δ=2 at the same ω are lower. Please rephrase to describe the non-monotonic trend.
  2. [Figure 2] The Figure 2 caption states “using ω=0 and δ=10,” but the main experimental default in §4.1 is δ=4. Please make the figure and text consistent and state which figure corresponds to which setting.
  3. [Abstract; throughout] “close-sourced” should be “closed-source.”
  4. [§3.4; Supplementary §1] The tangent-based schedule derivation is valid, but it is a reparameterization of the DDPM cosine schedule via SNR equality. Please cite any prior SNR-equivalence or log-SNR-based schedule formulations and temper the wording “emerges naturally” in the supplement.
  5. [§3.3] PSCA is described as adding “zero additional computational cost.” Adding log(M) to the attention logits is a negligible cost but not literally zero FLOPs; recommend “negligible overhead.”
  6. [§4.2, Table 6] Please define “zero-shot” explicitly (no training/fine-tuning on Movie Gen data) and note that FAD/KAD cannot be computed on Movie Gen Audio Bench because no ground-truth audio is available; this context is important for interpreting the IS-only comparison.
  7. [§4.3, Table 5] Please clarify whether the two rows use the same generated captions at inference and differ only in the prompt configuration used to produce them. If the inference captions also differ, state that explicitly.

Circularity Check

0 steps flagged

No load-bearing circularity: the tangent-based schedule is an explicit SNR-equating reparameterization, PSCA's omega/delta/CFG are disclosed tunings rather than fitted predictions, and the SOTA claims are benchmarked against external baselines with official checkpoints. The only self-citation is non-load-bearing.

full rationale

No circular steps found. The paper's only 'derived' result — the tangent-based sampling schedule t = 1 - 1/(tan(pi/2 u)+1) (Supp. Sec. 1; main Sec. 3.4) — is obtained by analytically equating the DDPM cosine SNR, SNR_DDPM(tau) = tan(pi/2(1-tau)), to the flow-matching SNR, SNR_FM(t) = t/(1-t), and solving for t. No model output or fitted value enters the derivation; it is a parameter-free reparameterization of a known cosine schedule, so the appropriate criticism is that it adds little novelty, not that it is circular. PSCA's aligned center frame jc(i) = floor(rv*i/ra) with omega=0 and delta=4 (Sec. 3.3, Sec. 4.1) is an architectural prior with disclosed hyperparameters, not a prediction; Table 2 varies omega/delta and shows Align Acc is non-monotonic (88.56, 88.37, 89.37, 88.02 for delta = 0, 2, 4, 8 at omega=0), so the metric is not forced by the mask by construction. The skeptic's concern that the hard linear window excludes off-screen or delayed sound sources is a robustness/correctness assumption about a design choice, not a circular reduction. The central SOTA claims are validated against independent benchmarks: baselines marked dagger are 'reproduced using the authors' official checkpoints and inference scripts,' MMAudio (diamond) is 'retrained on the VGGSound dataset using the official implementation,' and zero-shot results are on the externally constructed Movie Gen Audio Bench (Sec. 4.1, Table 6). The paper itself flags the only evaluation bias it sees (MDSGen and Frieren sharing the Align Acc backbone) rather than hiding it. The sole self-citation, ref. [48] (Van, Tran, Nguyen, Hy, ICASSP 2025), supports only the peripheral claim that text guidance benefits 'other domains' (Intro), and is not load-bearing for any architectural or SOTA claim. The stated limitations (Supp. Sec. 4: unintelligible speech, lower-quality music, out-of-distribution generalization) are data-bound admissions, not circularity. Overall, the derivation chain is self-contained; the score of 2 reflects only the presence of one minor, non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The paper’s positive contribution reduces to: (i) a hand-designed soft mask (ω, δ) with a progressive schedule β inserted into standard cross-attention, and (ii) a caption pipeline whose ground truth is video-SALMONN output. Everything else – the flow-matching path, VAE/BigVGAN codec, CLIP/FLAN-T5 encoders, MM-DiT/DiT blocks, CFG, data-noise alignment – is carried from cited prior work at zero marginal cost to the central claim. The load-bearing free choices are the mask geometry (ω=0, δ=4, linear β), λ=0.5, s=7.5, and the SNR-matching criterion that manufactures the tangent-based scheduler. No invented physical entities (no new forces, particles, or dimensions).

free parameters (5)
  • PSCA hard-attention window ω = 0
    Hand-set; Table 2 shows ω=0 maximizes Align Acc, and it is selected as the default (blue) setting.
  • PSCA fade-zone size δ = 4
    Selected from the sweep {0,2,4,8} in Table 2; stated as the default for all experiments.
  • Layer-progressive schedule βℓ = βℓ = 1 − ℓ/(N2−1)
    Linear decay from 1 to 0 chosen by hand; Table 4 shows removing it hurts, but no alternative schedule is tested.
  • Velocity-direction loss weight λ = 0.5
    Hand-set; no ablation reported for λ in the paper.
  • CFG guidance weight s = 7.5
    Tuned via the sweep in Supp. Fig. 4; chosen as the best KAD/IS balance.
axioms (6)
  • standard math Flow matching with linear interpolation xt = t·x1 + (1−t)·x0 and velocity vt = x1 − x0
    Section 3.1; taken from Lipman et al. [26] and Liu et al. [29].
  • domain assumption Audio-visual temporal correspondence is monotonic and frame-rate-linear: audio token i aligns with video frame jc(i)=floor(rv·i/ra), with hand-set tolerance ω, δ
    Section 3.3; the load-bearing premise of PSCA. Not learned or derived, and in tension with the off-screen sound events SoundCap claims to recover.
  • ad hoc to paper Equating DDPM-cosine SNR with flow-matching SNR is the correct design criterion for the training-time scheduler
    Supp. §1; this premise manufactures the tangent-based schedule (Eq. 5). The criterion is chosen, not derived, and no ablation against logit-normal is given.
  • domain assumption video-SALMONN’s generated captions are trustworthy enough to serve as ground truth for fine-tuning the VLM, without human verification at scale
    Section 3.5; only a competitor’s failure anecdote (car engine vs lion roar) is given, none for video-SALMONN.
  • domain assumption Frozen pretrained encoders (CLIP, FLAN-T5, VAE, BigVGAN) faithfully represent visual, textual, and audio content
    Section 3.2; standard practice for the field, but the VAE and CLIP variants are not fully specified.
  • ad hoc to paper The Align Acc deficit is explained by encoder overlap: Frieren/MDSGen use the same pretrained encoder as the CAVP metric backbone
    Section 4.2; used to discount the one metric where Flowley does not win. The overlap mechanism is asserted, not demonstrated quantitatively.

pith-pipeline@v1.3.0-alltime-deepseek · 22753 in / 24677 out tokens · 233111 ms · 2026-08-02T08:17:21.674152+00:00 · methodology

0 comments
read the original abstract

Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.

Figures

Figures reproduced from arXiv: 2607.06405 by Long-Khanh Pham, Luong Tran, Ngoc-Son Nguyen, Paarth Neekhara, Shehzeen Hussain, Thanh V. T. Tran, Van Nguyen.

Figure 1
Figure 1. Figure 1: (a) The proposed Flowley framework consists of two core modules. (b) First, visual, textual, and audio latent representations are processed together through the multi-stream block. (c) Latent features are then passed into the single-stream block, where they undergo weighted cross-attention with the visual and textual streams to estimate the flow field. At inference time, we integrate this learned flow usin… view at source ↗
Figure 2
Figure 2. Figure 2: Mask values within the fade region of the progressive soft-mask M across all N2 = 10 single-stream blocks, using ω = 0 and δ = 10. 0.0 0.2 0.4 0.6 0.8 1.0 t 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 (t) Tangent-based Logit-Normal [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of SoundCap’s ground-truth generation, training, and inference pipeline. Blue labels denote audio events that a standard VLM can detect, while red labels highlight events that are challenging to infer from visuals alone. labels (most under 6-10 words), limiting the semantic granularity available to the model. SoundCap remedies this limitation by: (1) leveraging an AV-LLM to craft fine-grained, sou… view at source ↗
Figure 5
Figure 5. Figure 5: Human preference comparison between Flowley and competing methods. as a powerful performance multiplier, most notably for MMAudio in the distri￾bution matching category. By incorporating these rich descriptions, MMAudio achieves the best FAD score of 7.09. Furthermore, SoundCap yields a significant 31.6% improvement in KAD for MMAudio. For Flowley, SoundCap also im￾proves overall performance across several… view at source ↗
Figure 1
Figure 1. Figure 1: Noise-aware (blue) prompt config￾uration for robust audio-visual LLM gen￾eration. ROLE: You are SoundCaptioner, a vision-language assistant. Your goal is to write sound-aware captions for silent videos by inferring the audible scene using only visual cues. CORE TASK: Produce a sound-aware caption for the silent video by writing exactly ONE concise English sentence. GUIDELINES: 1. Sound Identification & Des… view at source ↗
Figure 3
Figure 3. Figure 3: Noise-naive prompt configuration. 1 2 3 4 5 6 7 8 9 10 11 CFG Weight 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 KAD (the lower the better) 8 10 12 14 16 18 IS (the higher the better) [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of videos in Movie Gen Audio Bench that are well/not well covered by VGGSound. Top: Flowley achieves higher IB-Scores on videos featuring familiar concepts (birds with 2,508 samples and swimming with 516), which are well represented in the training data. Bottom: Performance drops for unfamiliar concepts (there are no videos related to mashed potatoes or rockets), which are absent from VGGSound, le… view at source ↗
Figure 6
Figure 6. Figure 6: Examples of sound-aware caption generated by SoundCap. Learning. Proceedings of Machine Learning Research, vol. 235, pp. 12606–12633. PMLR (21–27 Jul 2024), https://proceedings.mlr.press/v235/esser24a.html 5. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference on Computer Vision and P… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison on VGGSound. Flowley is the only method that ac￾curately identifies the drone-shot moment, correctly suppressing motor noise. It also captures the transition in camera angle with high precision [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison on VGGSound. Flowley, Frieren, and MMAudio are the only three methods capable of generating sounds at both distinct crowing moments. However, while the other two fail to accurately capture the timing of the louder, more prolonged crow, Flowley succeeds in doing so, demonstrating notably precise alignment [PITH_FULL_IMAGE:figures/full_fig_p029_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative comparison on VGGSound. In the fast-motion scenario, although Flowley does not produce audio that is perfectly aligned with the ground-truth, the generated output remains perceptually plausible. Notably, Flowley is the only method that precisely captures the instant when the person flips the switch, producing a subtle “click” sound [PITH_FULL_IMAGE:figures/full_fig_p030_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative comparison on VGGSound. In scenarios with repetitive events (a dancer repeatedly tapping his/her feet on the floor), Flowley still achieves the highest accuracy among all methods [PITH_FULL_IMAGE:figures/full_fig_p031_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 3 linked inside Pith

  1. [1]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 721–725 (2020).https://doi.org/ 10.1109/ICASSP40776.2020.9053174

    arXiv 2020

  3. [3]

    Cheng, H.K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A., Mitsufuji, Y.: Mmaudio: Taming multimodal joint training for high-quality video-to-audio syn- thesis.In:ProceedingsoftheComputerVisionandPatternRecognitionConference (CVPR). pp. 28901–28911 (June 2025)

    2025

  4. [4]

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Rom- bach, R.: Scaling rectified flow transformers for high-resolution image synthesis. In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp,F.(eds.)Proceed...

    2024

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15180–15190 (June 2023)

    2023

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., Laine, S.: Analyzing and improving the training dynamics of diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24174–24184 (June 2024)

    2024

  7. [7]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)

    Kushwaha, S.S., Tian, Y.: Vintage: Joint video and text conditioning for holistic audio generation. In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). pp. 13529–13539 (June 2025)

    2025

  8. [8]

    In: International Conference on Learning Representations (2019),https://openreview.net/forum? id=Bkg6RiCqY7

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019),https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  9. [9]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Luo, S., Yan, C., Hu, C., Zhao, H.: Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Informa- tion Processing Systems. vol. 36, pp. 48855–48876. Curran Associates, Inc. (2023),https : / / proceedings . neurips . cc / paper _ f...

    2023

  10. [10]

    In: Meila, M., Zhang, T

    Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 8 Tran et al. Frieren FoleyCrafter V2A-Mapper MDSGen Mel-QCD VinTAGe MMAudio Flowley (ours) Ground-truth drone-shot, no ...

    2021

  11. [11]

    In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=yFEqYwgttJ

    Pham, T.X., Ton, T., Yoo, C.D.: MDSGen: Fast and efficient masked diffu- sion temporal-aware transformers for open-domain sound generation. In: The Thirteenth International Conference on Learning Representations (2025),https: //openreview.net/forum?id=yFEqYwgttJ

    2025

  12. [12]

    arXiv preprint arXiv:2410.13720 (2024)

    Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.Y., Chuang, C.Y., et al.: Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    (eds.) Proceedings of the 41st International Conference on Machine Learning

    Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., Wang, Y., Zhang, C.: video-SALMONN: Speech-enhanced audio-visual large language models.In:Salakhutdinov,R.,Kolter,Z.,Heller,K.,Weller,A.,Oliver,N.,Scarlett, J., Berkenkamp, F. (eds.) Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Re...

    2024

  14. [14]

    Wang, H., Ma, J., Pascual, S., Cartwright, R., Cai, W.: V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. Proceed- ings of the AAAI Conference on Artificial Intelligence38(14), 15492–15501 (Mar 2024).https://doi.org/10.1609/aaai.v38i14.29475,https://ojs.aaai.org/ index.php/AAAI/article/view/29475

    doi:10.1609/aaai.v38i14.29475 2024

  15. [15]

    In: Proceed- ings of the Computer Vision and Pattern Recognition Conference (CVPR)

    Wang, J., Xu, C., Yu, C., Shang, L., Hu, Z., Wang, S., Bo, L.: Synchronized video- to-audio generation via mel quantization-continuum decomposition. In: Proceed- ings of the Computer Vision and Pattern Recognition Conference (CVPR). pp. 3111–3120 (June 2025)

    2025

  16. [16]

    In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C

    Wang, Y., Guo, W., Huang, R., Huang, J., Wang, Z., You, F., Li, R., Zhao, Z.: Frieren: Efficient video-to-audio generation network with recti- fied flow matching. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C. (eds.) Advances in Neural Informa- tion Processing Systems. vol. 37, pp. 128118–128138. Curran Associates...

    2024

  17. [17]

    Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., Chen, K.: Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds (2024),https: //arxiv.org/abs/2407.01494

    Pith/arXiv arXiv 2024