Pith. sign in

REVIEW 4 major objections 4 minor 48 references

A single trainable GPT-2 mapper, fed with fused features from two frozen visual encoders, can turn silent video into synchronized audio at a fraction of the usual training cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A GPT-2 mapper over dual visual encoders claims 16% training cost and better alignment, but test-time use of true class labels makes the comparison invalid for V2A.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid architecture, but test-time oracle labels and a biased training subset break the central efficiency claim. the 4 major comments →

arxiv 2509.04957 v1 pith:HU52PQWJ submitted 2025-09-05 cs.CV cs.MMcs.SDeess.AS

Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper

Gehui Chen , Guan'an Wang , Xiaowen Huang , Jitao Sang This is my paper
classification cs.CV cs.MMcs.SDeess.AS
keywords video-to-audio generationfoundation modelscross-modal mappingautoregressive GPT-2 mappertime-aligned feature fusionAudioLDM-2AudioMAE embeddingstraining efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MFM-Mapper is a video-to-audio system built by connecting frozen foundation models with a single trainable mapper, and the paper claims this makes video-to-audio training much cheaper without losing alignment. Instead of training a video-conditioned audio model from scratch, it fuses features from two pretrained visual encoders—CAVP, tuned for audio-visual synchronization, and TimeChat, tuned for video semantics and timing—then fine-tunes a GPT-2 language model to translate the fused visual features into the audio-conditioning embeddings used by AudioLDM-2. The reported result is that this pipeline, trained on 8,000 hours of video-audio pairs (16% of the 50,000 hours used by V2A-Mapper), improves semantic alignment (ImageBind score up 15.5%) and temporal sync (DeSync down 12.4%) relative to V2A-Mapper, and is competitive with models trained on much larger corpora. A reader should care because it suggests the bottleneck in video-to-audio is not data volume but how visual information is extracted and mapped into an existing audio generator.

Core claim

MFM-Mapper's claim is that a video-to-audio system can be built from frozen foundation models plus one trainable mapper, and that the gains come from how the visual features are chosen and fused. CAVP and TimeChat embeddings are upsampled to a common frame rate, concatenated, and linearly projected; a fine-tuned GPT-2 then autoregressively predicts pooled AudioMAE embeddings, the same condition AudioLDM-2 uses for synthesis. On VGGSound, MFM-Mapper reports higher ImageBind score and lower DeSync than V2A-Mapper with 16% of its training scale, and its 100-epoch variant also narrows the Fréchet-distance gap. Ablations show both encoders, time-aligned fusion, and the autoregressive mapper each

What carries the argument

Two mechanisms carry the argument. First, a fine-tuned GPT-2 serves as an autoregressive cross-modal mapper: it takes the fused visual sequence as input and, with teacher forcing and an MSE loss, predicts pooled AudioMAE embeddings—the conditioning target AudioLDM-2 was trained on. This reuses AudioLDM-2's own GPT-2 translation recipe but replaces text embeddings with visual embeddings. Second, time-aligned fusion brings the TimeChat embeddings up to CAVP's temporal resolution before concatenation, so the fused sequence keeps the video's frame-level timing. CAVP supplies audio-visual synchronization, TimeChat supplies semantics and temporal understanding, and the ablation tables isolate each

Load-bearing premise

The load-bearing premise is that supplying the ground-truth audio class label as a text prompt at test time is a legitimate video-to-audio condition; without those labels, the semantic-alignment gains over V2A-Mapper may shrink or disappear, and the 16%-scale efficiency claim is not established by a controlled comparison in the usual label-free video-to-audio setting.

What would settle it

Run MFM-Mapper on the VGGSound/VisualSound test set without providing the dataset's audio class labels as FLAN-T5 text prompts, keeping only the CAVP+TimeChat visual input. If the ImageBind score and DeSync over V2A-Mapper collapse toward or below the V2A-Mapper values, then the reported 16%-scale efficiency advantage depends on knowing the answer the model is supposed to produce.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Video-to-audio systems can be assembled by connecting frozen vision and audio foundation models, making the cost of a new system roughly the cost of training one GPT-2 mapper.
  • The 16% training-scale result implies that data efficiency in video-to-audio comes largely from feature extraction and mapping design, not from the size of the training corpus.
  • If the gains hold, a practical recipe for synchronized sound on silent video is now available: use a contrastive audio-visual encoder plus a temporal video-language encoder, fuse at a matched frame rate, and let a language model write the audio conditioning.
  • The 100-epoch variant suggests the same architecture can trade training time for better distribution matching while semantic and temporal alignment plateau early.
  • Researchers evaluating video-to-audio systems should report training scale and mapper parameters alongside quality metrics, as this paper does, because two systems with similar scores can differ by an order of magnitude in training cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the same dual-encoder plus GPT-2 mapper recipe could be transferred to other frozen audio generators, not just AudioLDM-2, making the mapping module reusable across audio backbones.
  • Because the test-time text prompts are ground-truth audio class labels (e.g. 'dog bow-wow'), the efficiency claim is strongest in settings where the semantic class is known in advance; a natural next experiment is the fully label-free video-to-audio setting where the visual encoders alone must supply semantics.
  • The reported plateau in semantic and temporal alignment around 40 epochs, while Fréchet distance keeps improving, suggests the training budget could be split into a short alignment phase and a longer quality-refinement phase.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MFM-Mapper, a video-to-audio (V2A) method that connects two frozen visual foundation models (CAVP and TimeChat) through a time-aligned fusion module to a fine-tuned GPT-2 mapper, which predicts pooled AudioMAE embeddings that condition the frozen AudioLDM-2 text-to-audio diffusion model. The authors claim that, compared to the previous mapper-based V2A-Mapper, MFM-Mapper achieves better semantic and temporal consistency while requiring only 16% of the training scale, and is competitive with much larger models. Experiments on VGGSound/VisualSound report IB-score and DeSync improvements over V2A-Mapper, plus ablations on visual encoders, fusion methods, and autoregressive versus diffusion mapping.

Significance. If the central claims were valid, the paper would make a useful contribution: a parameter-efficient, data-efficient recipe for V2A by reusing multiple frozen foundation models and a pretrained autoregressive mapper. The idea of fusing complementary visual encoders and using GPT-2 as a continuous mapper is technically reasonable and worth exploring. However, the experimental evidence as presented does not support the headline efficiency and semantic-consistency claims because of test-time oracle label injection and a confounded training-scale comparison. The paper's core architectural ideas are not invalidated, but they need a controlled evaluation before the claims can be accepted.

major comments (4)
  1. [Section 4.1.2] Test-time ground-truth audio class labels are provided as FLAN-T5 text prompts ('dog bow-wow') during inference. AudioLDM-2 is jointly conditioned on these text embeddings, and the authors state that their absence significantly affects audio quality. This gives MFM-Mapper privileged semantic information about the target audio that V2A-Mapper and other baselines do not receive. The reported IB-score improvement (26.07 vs 22.58, +15.5%) is therefore not attributable to the proposed architecture. A valid comparison must evaluate MFM-Mapper without ground-truth labels, or provide the same label information to all baselines.
  2. [Table 1 / Section 4.1.1] The training-scale comparison is uncontrolled. MFM-Mapper is trained on VisualSound, a subset of VGGSound (77k clips, ~40%) filtered by ImageBind for audio-visual coherence, whereas V2A-Mapper's 50,000h is presumably the full VGGSound training set. The 16% figure conflates dataset size, curation, and epochs. To support the data-efficiency claim, the authors should train a baseline mapper on the same VisualSound subset with the same number of epochs, or train MFM-Mapper on the full VGGSound training set, and compare under identical data and label conditions.
  3. [Sections 4.1.1 and 4.2.1] There is a circularity concern: ImageBind is used both to curate the VisualSound training subset and to compute the semantic evaluation metric (IB-score). Training on samples that score highly in ImageBind's embedding space is likely to inflate IB-score independent of actual V2A quality. The authors should report an independent semantic metric (e.g., CLAP score or audio classification accuracy) and/or a metric not derived from the same embedding model.
  4. [Section 4.4.3 / Figure 3] The comparison between the proposed autoregressive mapper and the diffusion-based 'Diff-Mapper' is not fully described. It is unclear whether Diff-Mapper also receives the same text-label conditioning at test time, and whether hyperparameters (learning rate, epochs, batch size) are matched. Without this control, the faster convergence claim in Figure 3 is not conclusive. Please clarify or add the missing protocol details.
minor comments (4)
  1. [Section 4.4.2 / Table 3] The text states that the 'Cat' method shows lower performance on 'Acc', but there is no 'Acc' metric in Table 3. This appears to be a typo; likely 'DeSync' or 'IB-score' is intended.
  2. [Section 3.4] In the teacher-forcing description, the model's own predictions are written as 'c1, c1,..., c1' instead of c-hat subscripts. Please correct the notation.
  3. [Abstract and Section 1] The phrase 'requiring only 16% of the training scale' is imprecise because training scale is defined later as (dataset hours) x (epochs). Since the dataset is also different, '16%' should be accompanied by a clear definition and the actual hours/epochs of each method.
  4. [Section 3.2.2] Algorithm 1 says 'Upsample' but the caption of Figure 1 says 'replicated'. Clarify whether upsampling is nearest-neighbor replication or another interpolation method.

Circularity Check

1 steps flagged

Test-time ground-truth class labels as FLAN-T5 prompts give MFM-Mapper the semantic answer, making the headline semantic-consistency gain over V2A-Mapper an oracle-conditioned result rather than a video-to-audio prediction.

specific steps
  1. other [Section 4.1.2, Implementation details (arXiv:2509.04957v1, p.8)]
    "Preliminary experiments have revealed that the absence of FLAN-T5 embeddings significantly impacts audio quality. To fully leverage its capabilities, during test-time inference, we also provide audio class labels from the dataset as text prompts, such as 'dog bow-wow'. These labels are encoded into text embeddings by FLAN-T5 and used as the second conditional input to the diffusion model."

    The headline semantic-consistency result (IB-score 26.07 vs 22.58, +15.5%) is attributed to MFM-Mapper's visual encoders and GPT-2 mapper. But at inference the model is given the ground-truth audio class label as a text prompt to AudioLDM-2, while V2A-Mapper is not. AudioLDM-2 is a text-conditioned generator, so 'dog bow-wow' already fixes the semantic category of the output; the IB-score then measures agreement with the very label supplied as input. The comparison is therefore not a video-to-audio semantic prediction but text-to-audio generation conditioned on the target class. The reported semantic improvement is forced by the experimental protocol, not by the proposed mapping.

full rationale

The core architecture is not circular: dual frozen visual encoders, time-aligned fusion, a fine-tuned GPT-2 mapper regressing to pooled AudioMAE embeddings, and a frozen AudioLDM-2 are a legitimate pipeline whose training target (ground-truth AudioMAE) is external to the evaluation metrics. The decisive circular element is the test-time ground-truth class-label prompt, which injects the semantic answer into the generator and then claims the semantic-consistency metric as evidence for the video-to-audio mapper. This makes the central 'better semantic consistency' claim partially reduce to an input rather than to the method. The ImageBind-filtered VisualSound training subset combined with ImageBind-based IB-score is a related confound, but since evaluation uses the full VGGSound test set, it is not a by-construction circularity; it does, however, undermine the 16%-scale efficiency comparison because the two training sets are not matched. The 16% figure itself is arithmetic (8,000h / 50,000h) and not circular. Temporal and audio-quality metrics (DeSync, FD, IS) rely on external models and are not part of the circular chain.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 1 invented entities

The central claim rests on multiple hand-chosen hyperparameters and domain assumptions, plus one ad hoc assumption (test-time label availability) that is load-bearing and unrealistic for the stated V2A task. The model architecture itself is a new ledger entry with no released code or independent verification.

free parameters (9)
  • Test-time text prompts = ground-truth audio class labels (e.g., 'dog bow-wow')
    The paper conditions AudioLDM-2 on FLAN-T5 text embeddings of the dataset's class labels during inference, providing privileged semantic information not available in standard V2A and not used by baselines (Section 4.1.2).
  • Training epochs = 40 (MFM-Mapper), 100 (MFM-Mapper-M)
    The 40-epoch run defines the headline 16% training scale; the 100-epoch run performs better on FD but not on IB/DeSync, so the reported efficiency claim depends on this choice.
  • Learning rate = 1e-3
    AdamW LR taken from AudioLDM-2 config; chosen by hand.
  • CFG scale = 3.5
    Classifier-free guidance strength for AudioLDM-2 inference; chosen following AudioLDM-2 best practices, affects quality and alignment.
  • Denoising steps = 100
    DDIM inference steps for AudioLDM-2; set to 100 as recommended.
  • Negative prompt = 'low quality,average quality'
    Negative prompt for CFG; fixed by hand and used for all samples.
  • Frame sampling rate = 4 FPS
    Video frames are sampled at 4 FPS for both encoders; this temporal resolution affects the alignment granularity.
  • Fusion method = channel-wise concatenation + linear projection
    Chosen among additive and time-dimension concatenation based on the ablation in Table 3; a design decision that affects all downstream results.
  • GPT-2 configuration = 12 layers, 768 hidden dim, initialized from AudioLDM-2's released weights
    The mapper architecture and initialization are borrowed from AudioLDM-2; this choice is load-bearing for the 'pre-trained mapper' claim.
axioms (6)
  • domain assumption Frozen foundation models can be connected by a learned mapper without finetuning the generators.
    This is the core assumption of the mapper-based V2A line (V2A-Mapper), inherited and extended. It is not re-derived in this paper.
  • domain assumption Pooled AudioMAE embeddings are an adequate condition space for temporally coherent audio generation.
    The paper relies on AudioLDM-2's design that pooled AudioMAE carries temporal structure (Section 3.3). This is an upstream assumption, not tested here.
  • domain assumption The VisualSound subset (ImageBind-filtered VGGSound) is representative enough of the full VGGSound test distribution.
    The model trains only on VisualSound (~77k videos filtered by ImageBind) but evaluates on the complete VGGSound test set (Section 4.1.1). No analysis of domain shift or class coverage is given.
  • ad hoc to paper Ground-truth audio class labels are available at test time.
    The method uses dataset class labels as text prompts at inference (Section 4.1.2). This assumption is not stated as a limitation and is invalid for arbitrary silent video.
  • standard math Minimizing MSE between predicted and ground-truth pooled AudioMAE features maximizes the autoregressive likelihood in Equation (1).
    This holds only under a fixed-variance Gaussian model; the paper states it as exact without derivation (Section 3.4).
  • domain assumption Teacher forcing with ground-truth context during training transfers to autoregressive decoding at inference.
    Standard practice from AudioLDM-2; the paper does not address exposure bias.
invented entities (1)
  • MFM-Mapper architecture no independent evidence
    purpose: Map time-aligned fused visual features (CAVP + TimeChat) to pooled AudioMAE conditions for AudioLDM-2
    The new neural network is the paper's contribution, but no code or weights are released, so the only evidence is self-reported metrics on VGGSound; there is no external falsifiable handle beyond the paper's own experiments.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper." pith.science (2026). https://pith.science/paper/HU52PQWJ

@misc{pith2026250904957,
  author       = {Pith},
  title        = {Pith review of: Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HU52PQWJ}},
  note         = {Machine review of arXiv:2509.04957}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent Video-to-Audio (V2A) generation relies on extracting semantic and temporal features from video to condition generative models. Training these models from scratch is resource intensive. Consequently, leveraging foundation models (FMs) has gained traction due to their cross-modal knowledge transfer and generalization capabilities. One prior work has explored fine-tuning a lightweight mapper network to connect a pre-trained visual encoder with a text-to-audio generation model for V2A. Inspired by this, we introduce the Multiple Foundation Model Mapper (MFM-Mapper). Compared to the previous mapper approach, MFM-Mapper benefits from richer semantic and temporal information by fusing features from dual visual encoders. Furthermore, by replacing a linear mapper with GPT-2, MFM-Mapper improves feature alignment, drawing parallels between cross-modal features mapping and autoregressive translation tasks. Our MFM-Mapper exhibits remarkable training efficiency. It achieves better performance in semantic and temporal consistency with fewer training consuming, requiring only 16\% of the training scale compared to previous mapper-based work, yet achieves competitive performance with models trained on a much larger scale.

Figures

Figures reproduced from arXiv: 2509.04957 by Gehui Chen, Guan'an Wang, Jitao Sang, Xiaowen Huang.

Figure 1
Figure 1. Figure 1: Overall framework of MFM-Mapper. Input video frames are processed by two frozen visual FMs (CAVP and TimeChat). The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative comparison of generated audio spectrograms. Each panel displays video frames, the ground-truth spectrogram, and [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics comparing the MFM-Mapper (Ours) and the Diff-Mapper over 100 epochs on key alignment metrics. (a) [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 34 canonical work pages · 2 internal anchors

  1. [1]

    Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, and Syed Sameed Husain. 2024. Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification.arXiv preprint arXiv:2408.14441(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators.OpenAI(2024). https://openai.com/research/video-generation- models-as-world-simulators Accessed: 2025-03-08

    work page 2024

  3. [3]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. VGGSound: A Large-scale Audio-Visual Dataset. InICASSP

    work page 2020

  4. [4]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478(2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. 2025. Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis. InCVPR

    work page 2025

  6. [6]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    Pith/arXiv arXiv 2022

  7. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning.ArXiv preprint arXiv:2305.06500(2023)

    Pith/arXiv arXiv 2023

  8. [8]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. InICLR. https://openreview.net/forum?id=YicbFdNTTy

    work page 2021

  9. [9]

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. 2023. Conditional generation of audio from video via foley analogies. In CVPR

    work page 2023

  10. [10]

    Jort F Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. InICASSP

    work page 2017

  11. [11]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. ImageBind: One Embedding Space To Bind Them All. InCVPR

    2023

  12. [12]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. InNeurIPS

    2020

  13. [13]

    Jonathan Ho and Tim Salimans. 2022. Classifier-Free Diffusion Guidance.arXiv preprint arXiv:2207.12598(2022)

    Pith/arXiv arXiv 2022

  14. [14]

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. Masked Autoencoders that Listen. InNeurIPS

    work page 2022

  15. [15]

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. 2024. Synchformer: Efficient synchronization from sparse cues. InICASSP

    work page 2024

  16. [16]

    Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2019. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. InInterspeech

    work page 2019

  17. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR)

    work page 2015

  18. [18]

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition.IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)(2020)

    work page 2020

  19. [19]

    Khaled Koutini, Jan Schlüter, Hamid Eghbal-Zadeh, and Gerhard Widmer. 2022. Efficient training of audio transformers with patchout. InInterspeech

    work page 2022

  20. [20]

    Junwon Lee, Jaekwon Im, Dabin Kim, and Juhan Nam. 2024. Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound.arXiv preprint arXiv:2408.11915(2024)

    arXiv 2024

  21. [21]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InICML

    work page 2023

  22. [22]

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wen Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding.ArXiv preprint arXiv:abs/2305.06355(2023)

    Pith/arXiv arXiv 2023

  23. [23]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747(2022). Manuscript submitted to ACM 14 G.Chen, G.Wang, X.Huang et al

    Pith/arXiv arXiv 2022

  24. [24]

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. InICML

    work page 2023

  25. [25]

    Plumbley

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. 2024. AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining.IEEE/ACM Transactions on Audio, Speech, and Language Processing32 (2024), 2871–2883. doi:10.1109/TASLP.2024.3399607

    work page arXiv 2024

  26. [26]

    Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023. Flow straight and fast: Learning to generate and transfer data with rectified flow. InICLR

    work page 2023

  27. [27]

    Xiulong Liu, Kun Su, and Eli Shlizerman. 2024. Tell what you hear from what you see — video to audio generation through text. InNeurIPS

    work page 2024

  28. [28]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR)

    work page 2019

  29. [29]

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. 2023. Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models. In NeurIPS. https://openreview.net/forum?id=q5FAZAIooz

    work page 2023

  30. [30]

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. 2023. Foleygen: Visually-guided audio generation.arXiv preprint arXiv:2309.10537(2023)

    Pith/arXiv arXiv 2023

  31. [31]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. InICCV

    work page 2023

  32. [32]

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, and et al. 2024. Movie Gen: A Cast of Media Foundation Models.arXiv preprint arXiv:2410.13720(2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InICLR

    work page 2021

  34. [34]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI(2019). https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Accessed: 2025-03-08

    work page 2019

  35. [35]

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding. InCVPR

    work page 2024

  36. [36]

    Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu. 2024. Sta-v2a: Video-to-audio generation with semantic and temporal alignment.arXiv preprint arXiv:2409.08601(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models . InCVPR

    work page 2022

  38. [38]

    Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. InICLR

    work page 2020

  39. [39]

    Stilwell

    Robynn J. Stilwell. 2007. The fantastical gap between diegetic and nondiegetic

    work page 2007

  40. [40]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InNeurIPS

    work page 2017

  41. [41]

    Ilpo Viertola, Vladimir Iashin, and Esa Rahtu. 2024. Temporally Aligned Audio for Video with Autoregression.arXiv preprint arXiv:2409.13689 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. 2024. V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models. InAAAI

    work page 2024

  43. [43]

    Xihua Wang, Yuyue Wang, Yihan Wu, Ruihua Song, Xu Tan, Zehua Chen, Hongteng Xu, and Guodong Sui. 2024. TiVA: Time-Aligned Video-to-Audio Generation. InACM MM. https://openreview.net/forum?id=tIpOYtxerl

    work page 2024

  44. [44]

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. 2024. Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching. InCVPR

    work page 2024

  45. [45]

    Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. InICASSP

    work page 2023

  46. [46]

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. 2024. Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners. InCVPR

    work page 2024

  47. [47]

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. 2024. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds.arXiv preprint arXiv:2407.01494(2024)

    Pith/arXiv arXiv 2024

  48. [48]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.arXiv preprint arXiv:2304.10592(2023). Manuscript submitted to ACM

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.