Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Smooth-Foley claims that frame-wise visual conditions plus text-label-guided temporal signals keep video-to-audio generation continuous for moving sound sources, outperforming Diff-Foley and FoleyCrafter on continuous-sound benchmarks.

desk verdict A sensible idea for a real V2A failure mode, but the paper's own ablation leaves the mechanism underdetermined; worth reviewing, not worth taking on faith. read the letter →

arxiv 2412.18157 v1 pith:3HIUAJLZ submitted 2024-12-24 cs.SD cs.AIeess.AS

Yaoyun Zhang , Xuenan Xu , Mengyue Wu This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords video-to-audiogenerationFoleysoundsynthesissemanticguidancetemporalalignmentdiffusionmodeladaptationCLIPsimilaritycontinuousVGGSound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a specific failure of video-to-audio generation: models go silent or lose synchronization when the sound source moves, becomes small, or leaves the frame. The proposed Smooth-Foley feeds the generator two cleaner conditions—per-frame CLIP image embeddings instead of one clip-level embedding, and a temporal on/off signal obtained by comparing each frame to the text label with CLIP and thresholding at 0.5. Both conditions enter through lightweight adapters into a frozen text-to-audio diffusion model, preserving pre-trained audio quality while sharpening video conditioning. The authors also filter a continuous-sound subset, VGGSound-Continuous, and fine-tune on it. They report that on VGGSound and on this subset, Smooth-Foley beats Diff-Foley and FoleyCrafter on objective metrics and human ratings, and that its outputs follow physical patterns such as Doppler-like frequency shifts.

What carries the argument

Two trainable adapters attached to a frozen pre-trained text-to-audio diffusion model (Auffusion). The frame adapter projects per-frame CLIP image embeddings through a linear layer and injects them via parallel cross-attention, keeping only the frame key/value projection matrices trainable. The temporal adapter reuses ControlNet's UNet-encoder design, is trained on AudioSet-strong with ground-truth timestamp conditions, and at inference consumes a binarized temporal condition: cosine similarity between projected frame CLIP embeddings and the label's CLIP text embedding, thresholded at 0.5. Training the two adapters separately keeps the backbone frozen, so the model retains pre-trained audio quality while adding video conditioning.

What would settle it

Take a VGGSound-Continuous clip in which the label's sound source is off-screen or visually tiny for part of the clip, such as an ambulance siren heard before the ambulance enters the frame. Compute per-frame CLIP similarities between the frames and the label text; if the audible portion has similarities below 0.5 while the inaudible portion has similarities above it, the temporal condition is inverted, and Smooth-Foley should generate sound at the wrong times. If it does not fail on such a clip, the thresholding is not the mechanism driving temporal alignment.

Watch

Extended reading notes

Core claim

The central claim is that temporal alignment failures in video-to-audio generation stem from two correctable defects—low temporal resolution of video features and inaccurate temporal conditions—and that replacing clip-wise video embeddings with frame-wise CLIP embeddings, plus deriving the temporal condition from binarized frame-label CLIP similarities, fixes both. On VGGSound and on the filtered VGGSound-Continuous subset, Smooth-Foley reports lower FAD and mean KL divergence, higher CLIP scores, and better human ratings for semantic alignment, temporal alignment, and audio quality than Diff-Foley and FoleyCrafter. Qualitatively, it maintains sound during continuous motion, detects sound when the object becomes visually ambiguous, and produces frequency rises and falls consistent with the Doppler effect. The paper interprets these results as evidence that semantic guidance from textual labels yields more temporally aligned and physically plausible audio.

Load-bearing premise

The load-bearing premise is that a visual-text similarity score (CLIP) above 0.5 between a frame and the label reliably marks when the sound is actually audible, and that this same threshold works for every clip.

Editorial extensions

If this is right

  • For videos with moving or visually ambiguous sound sources, generated audio should remain present and synchronized instead of going silent.
  • The same frozen text-to-audio backbone can be adapted to video-to-audio generation with only lightweight adapters, avoiding retraining the audio generator from scratch.
  • Frame-wise visual features raise the effective temporal resolution of semantic video conditions beyond clip-level features, improving onset accuracy.
  • Using the textual label as a semantic guide for temporal-condition extraction makes the temporal signal more robust when the sounding object is small, blurred, or off-center.
  • On continuous-sound benchmarks such as VGGSound-Continuous, the model's outputs should exhibit higher audio quality, semantic alignment, and temporal alignment than Diff-Foley and FoleyCrafter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the 0.5 threshold is global, per-category thresholds or a learned detector could further improve off-screen or visually weak sound sources.
  • Editorial inference: the binarized CLIP similarity could be replaced by soft weights, letting the generator interpolate onset and offset instead of committing to hard on/off states.
  • Editorial inference: the VGGSound-Continuous filtering recipe (label-based selection plus text-to-audio grounding) could be reused as a public benchmark for continuous-sound video-to-audio evaluation.
  • Editorial inference: since the frame adapter and temporal adapter are trained separately, joint training could resolve potential disagreements between visual and temporal conditions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Smooth-Foley, a video-to-audio (V2A) generation model that adapts a pre-trained text-to-audio diffusion model (Auffusion) using two lightweight adapters: a frame adapter that injects frame-wise CLIP visual embeddings through parallel cross-attention (Section II.B, Eq. 2), and a temporal adapter that derives a temporal condition by binarizing frame-label CLIP similarities with a fixed threshold of 0.5 (Section II.C). The model is initialized from FoleyCrafter and fine-tuned on a filtered VGGSound-Continuous subset (Section II.D). The authors report that Smooth-Foley outperforms Diff-Foley and FoleyCrafter on MKL, CLIP Score, FAD, and subjective ratings on both VGGSound-Continuous and VGGSound, and they claim improved temporal alignment and adherence to physical laws such as the Doppler effect.

Significance. The paper addresses a real limitation of existing V2A systems: maintaining synchronized audio when the sound source is continuous or becomes visually ambiguous. The design is concrete and testable: frame-wise features and label-derived temporal conditions are well-motivated, and the use of a lightweight adapter on a frozen T2A model is practical. The authors report several standard metrics and include one ablation, which is a useful starting point. However, as presented, the experiments do not isolate the proposed semantic-guidance mechanism from additional fine-tuning data, from the oracle text label, or from the hand-set CLIP threshold. The core contribution therefore remains plausible but not established. If follow-up experiments with a temporal-adapter ablation, controlled baselines, and statistical evaluation confirm the reported gains, the method would be a solid incremental contribution to the V2A literature.

major comments (4)
  1. [III.B, Tables I and II] The only component-level ablation tests the frame adapter, and it shows that this component contributes essentially nothing to the headline metrics: on VGG-C, MKL is 1.558 with frame-wise features versus 1.559 without, and CLIP Score is 55.124 versus 55.076; on VGG, MKL is 2.515 versus 2.498 (slightly better without frame-wise), and the FAD comparisons are mixed (e.g., PANNs FAD on VGG is 18.31 with frame-wise versus 13.07 without). The proposed temporal adapter, which is the other half of the semantic guidance, is never ablated. Consequently, Tables I and II cannot attribute the reported superiority over FoleyCrafter and Diff-Foley to the proposed mechanism, as opposed to the extra fine-tuning on VGGSound-Continuous or to the oracle text label. I request an ablation that removes the temporal adapter and, ideally, an experiment that adds the same label-derived temporal condition to a baseline.
  2. [II.C, Table III] The central temporal-alignment claim is supported only by 10 subjective ratings per model and by qualitative examples in Figure 4; there is no event-boundary metric such as onset/offset accuracy or temporal IoU. The binarization threshold of 0.5 on CLIP frame-label similarities is asserted without justification or sensitivity analysis, and it is not validated against ground-truth sound event intervals. Because the temporal adapter can only be as good as its input condition, the paper needs either a direct evaluation of the estimated temporal conditions against AudioSet-strong-style annotations or an ablation with a different threshold (including a non-binarized similarity) to show that the threshold choice is not responsible for the results.
  3. [II.D, III.A] The comparison is confounded by training protocol. Smooth-Foley is initialized from FoleyCrafter and fine-tuned on VGGSound-Continuous (Section II.D), while the FoleyCrafter and Diff-Foley baselines appear to be evaluated without this fine-tuning. Any improvement on VGGSound-Continuous could therefore come from exposure to the target-domain data rather than from the frame adapter or label-guided temporal adapter. A controlled comparison would fine-tune the baselines on the same subset, or evaluate Smooth-Foley without fine-tuning, to separate the mechanism from the training data. The same concern applies to the 95-clip manually selected test set, which is not accompanied by error bars or significance tests; the reported differences in Table III (e.g., 8.42 vs. 5.89 for semantic alignment) need confidence intervals or a paired test to be interpretable.
  4. [Abstract and III.B.2] The claim that generated audio 'exhibits higher quality and better adherence to physical laws' (Abstract; also the Doppler-effect discussion in Section III.B.2) is supported only by a qualitative example. The Doppler claim in particular is stated as a frequency rise-peak-decline pattern but no spectrogram, pitch track, or quantitative comparison against ground truth is provided. If physical plausibility is part of the contribution, it should be measured; otherwise the statement should be softened.
minor comments (5)
  1. [II.B, Eq. (2)] The mixing weight lambda in Eq. (2) is never specified; please report its value or state how it was chosen. Also, the denominator uses a generic d without defining the dimension of K.
  2. [II.C] Please clarify the binarization operation: are frame similarities above 0.5 set to 1 and below to 0, and is the condition then used as a ControlNet-style conditioning map for the temporal adapter? The text says the temporal adapter is trained on AudioSet-strong with ground-truth timestamps but does not explain how the training condition is represented (e.g., binary mask, per-frame probability, event embedding).
  3. [III.A.2] The phrase 'Frechect Audio Distance' should be 'Fréchet Audio Distance'; also, the subjective evaluation uses only 10 samples per model, so please state whether the same 10 videos were used across models and report per-sample variance or inter-rater agreement.
  4. [Figure 4] The qualitative examples in Figure 4 would be much more informative with aligned spectrograms or waveform onset markers; as printed, the 'miss/✓' annotations are hard to verify without audio or time-aligned evidence.
  5. [II.D] The filtering procedure relies on text-to-audio grounding [22] but no details are given about the grounding model or the filtering threshold; please provide the number of clips after each filtering stage and the overlap between the 95 test clips and the training subset to avoid train/test leakage.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity in the derivation chain; the central V2A claims are externally benchmarked, with only a minor non-load-bearing self-citation in dataset filtering.

full rationale

The derivation chain is self-contained. The frame adapter (Eqs. 1-3) and the temporal adapter are trained with the standard diffusion objective, and the temporal condition is obtained by binarizing CLIP frame-label similarities at a fixed threshold of 0.5 (Sec. II.C), which is a hand-set hyperparameter rather than a parameter fitted to the test set. The central comparisons in Tables I-III are against external baselines (Diff-Foley, FoleyCrafter) on shared test splits, so the reported superiority is not equivalent to any input by construction. The only in-paper ablation (frame-wise vs. no-frame-wise) does not isolate the temporal adapter, and temporal alignment is supported mainly by 10-sample subjective ratings and qualitative examples; these are experimental-rigor limitations, not circular reasoning. The use of the authors' own text-to-audio grounding method [21,22] to filter VGGSound-Continuous is a tooling self-citation; it shapes the test set but does not force the model's output or the comparison outcome, so it is not load-bearing circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two hand-set hyperparameters (0.5 threshold and mixing weight) and on transfer assumptions about CLIP and AudioSet-strong. No new physical entities are introduced.

free parameters (2)
  • temporal condition binarization threshold = 0.5
    Used in Section II.C to convert CLIP frame-label similarities into binary temporal conditions; chosen by hand and not justified or swept in experiments.
  • mixing weight lambda for frame attention = not specified
    Appears in Eq. (2) and balances text and frame cross-attention; no value or sensitivity analysis is given.
assumptions (3)
  • domain assumption CLIP image-text similarity is a valid proxy for presence and timing of sound events in video frames.
    Central to the label-guided temporal condition in Section II.C; no validation against ground truth event timestamps is provided.
  • domain assumption The pretrained Auffusion T2A model can be adapted to V2A by parallel cross-attention adapters without harming generation quality.
    Adopted from FoleyCrafter [11], used throughout Section II.B.
  • domain assumption Temporal adapters trained on AudioSet-strong timestamps transfer to VGGSound videos without retraining on VGGSound timestamps.
    Section II.C trains the temporal adapter on AudioSet-strong but applies it to VGGSound during inference; distribution shift is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance." pith.science (2026). https://pith.science/paper/3HIUAJLZ

@misc{pith2026241218157,
  author       = {Pith},
  title        = {Pith review of: Smooth-Foley: Creating Continuous Sound for Video-to-Audio Generation Under Semantic Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HIUAJLZ}},
  note         = {Machine review of arXiv:2412.18157}
}
read the original abstract

The video-to-audio (V2A) generation task has drawn attention in the field of multimedia due to the practicality in producing Foley sound. Semantic and temporal conditions are fed to the generation model to indicate sound events and temporal occurrence. Recent studies on synthesizing immersive and synchronized audio are faced with challenges on videos with moving visual presence. The temporal condition is not accurate enough while low-resolution semantic condition exacerbates the problem. To tackle these challenges, we propose Smooth-Foley, a V2A generative model taking semantic guidance from the textual label across the generation to enhance both semantic and temporal alignment in audio. Two adapters are trained to leverage pre-trained text-to-audio generation models. A frame adapter integrates high-resolution frame-wise video features while a temporal adapter integrates temporal conditions obtained from similarities of visual frames and textual labels. The incorporation of semantic guidance from textual labels achieves precise audio-video alignment. We conduct extensive quantitative and qualitative experiments. Results show that Smooth-Foley performs better than existing models on both continuous sound scenarios and general scenarios. With semantic guidance, the audio generated by Smooth-Foley exhibits higher quality and better adherence to physical laws.

Figures

Figures reproduced from arXiv: 2412.18157 by the authors.

Figure 1
Figure 1. Examples to clarify the deficits. In the context of continuous sound and ambiguous object, FoleyCrafter (FC) fails to predict the presence of sound. In the first case, a train stays in the video for a long time but FC predicts event probabilities of nearly zero for some frames. In the second case, when flying airplane becomes tiny, FC fails to detect it, leading to unsatisfactory generation results. lead to more tem… view at source ↗
Figure 2
Figure 2. Overall pipeline of Smooth-Foley. Note that frame adapter and temporal controller module are trained separately. II. SMOOTH-FOLEY As previously described, Smooth-Foley integrates pre￾trained T2A models by lightweight adapters. Auffusion [12] is chosen as the T2A model, enabling adaptation to data-scarcity scenarios while keeping high-fidelity and diverse audio syn￾thesis abilities. As shown in [PITH_FULL_IMAGE:figu… view at source ↗
Figure 3
Figure 3. Label distribution of VGGSound-Continuous, most are sounding [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on temporal alignment with different models, i.e. FoleyCrafter (FC), Diff-Foley (DF) and Smooth-Foley (ours). [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages

  1. [1]

    Generating visually aligned sound from videos,

    P. Chen, Y . Zhang, M. Tan, H. Xiao, D. Huang, and C. Gan, “Generating visually aligned sound from videos,” IEEE Transactions on Image Processing, vol. 29, pp. 8292–8302, 2020

    work page 2020

  2. [2]

    Taming visually guided sound generation,

    V . Iashin and E. Rahtu, “Taming visually guided sound generation,” in British Machine Vision Conference , 2021

    work page 2021

  3. [3]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  4. [4]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  5. [5]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180–15 190

    2023

  6. [6]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  7. [7]

    Conditional generation of audio from video via foley analogies,

    Y . Du, Z. Chen, J. Salamon, B. Russell, and A. Owens, “Conditional generation of audio from video via foley analogies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2426–2436

    work page 2023

  8. [8]

    Varietysound: Timbre-controllable video to sound generation via unsupervised information disentanglement,

    C. Cui, Z. Zhao, Y . Ren, J. Liu, R. Huang, F. Chen, Z. Wang, B. Huai, and F. Wu, “Varietysound: Timbre-controllable video to sound generation via unsupervised information disentanglement,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    work page 2023

Show all 32 references
  1. [9]

    Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models,

    S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  2. [10]

    I hear your true colors: Image guided audio generation,

    R. Sheffer and Y . Adi, “I hear your true colors: Image guided audio generation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    2023

  3. [11]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,

    Y . Zhang, Y . Gu, Y . Zeng, Z. Xing, Y . Wang, Z. Wu, and K. Chen, “Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,” arXiv preprint arXiv:2407.01494 , 2024

    2024 arXiv

  4. [12]

    Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation,

    J. Xue, Y . Deng, Y . Gao, and Y . Li, “Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation,” arXiv preprint arXiv:2401.01044, 2024

    2024 arXiv

  5. [13]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,

    H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,” arXiv preprint arXiv:2308.06721 , 2023

    2023 arXiv

  6. [14]

    Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,

    D.-H. Lee et al., “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in Workshop on challenges in representation learning, ICML , vol. 3, no. 2. Atlanta, 2013, p. 896

    2013

  7. [15]

    Predicting deep zero-shot con- volutional neural networks using textual descriptions,

    J. Lei Ba, K. Swersky, S. Fidler et al., “Predicting deep zero-shot con- volutional neural networks using textual descriptions,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 4247– 4255

    2015

  8. [16]

    Integrating language guidance into vision-based deep metric learning,

    K. Roth, O. Vinyals, and Z. Akata, “Integrating language guidance into vision-based deep metric learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 177–16 189

    2022

  9. [17]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXiv preprint arXiv:2301.12503 , 2023

    2023 arXiv

  10. [18]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836–3847

    2023

  11. [19]

    The benefit of temporally-strong labels in audio event classification,

    S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The benefit of temporally-strong labels in audio event classification,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 366–370

    2021

  12. [20]

    Vggsound: A large- scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large- scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 721–725

    2020

  13. [21]

    Text-to-audio grounding: Build- ing correspondence between captions and sound events,

    X. Xu, H. Dinkel, M. Wu, and K. Yu, “Text-to-audio grounding: Build- ing correspondence between captions and sound events,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 606–610

    2021

  14. [22]

    Towards weakly supervised text-to- audio grounding,

    X. Xu, Z. Ma, M. Wu, and K. Yu, “Towards weakly supervised text-to- audio grounding,” arXiv preprint arXiv:2401.02584 , 2024

    2024 arXiv

  15. [23]

    Fr \’echet audio distance: A metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr \’echet audio distance: A metric for evaluating music enhancement algorithms,” arXiv preprint arXiv:1812.08466, 2018

    2018 arXiv

  16. [24]

    Correlation of fr \’echet audio distance with human perception of environmental audio is embedding dependant,

    M. Tailleur, J. Lee, M. Lagrange, K. Choi, L. M. Heller, K. Imoto, and Y . Okamoto, “Correlation of fr \’echet audio distance with human perception of environmental audio is embedding dependant,” arXiv preprint arXiv:2403.17508, 2024

    2024 arXiv

  17. [25]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  18. [26]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu*, K. Chen*, T. Zhang*, Y . Hui*, T. Berg-Kirkpatrick, and S. Dub- nov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP , 2023

    2023

  19. [27]

    Cnn architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold et al. , “Cnn architectures for large-scale audio classification,” in 2017 ieee international conference on acoustics, speech and signal processing (ICA...

    2017

  20. [28]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,

    H. Wang, J. Ma, S. Pascual, R. Cartwright, and W. Cai, “V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 492–15 501

    2024

  21. [29]

    Frieren: Efficient video-to-audio generation with rectified flow matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation with rectified flow matching,” arXiv preprint arXiv:2406.00320 , 2024

    2024 arXiv

  22. [30]

    Video-foley: Two-stage video-to- sound generation via temporal event condition for foley sound,

    J. Lee, J. Im, D. Kim, and J. Nam, “Video-foley: Two-stage video-to- sound generation via temporal event condition for foley sound,” arXiv preprint arXiv:2408.11915, 2024

    2024

  23. [31]

    Wav2clip: Learning robust audio representations from clip,

    H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 4563–4567

    2022

  24. [32]

    Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr ´echet audio distance: A reference-free metric for evaluating music enhancement algorithms,” in Interspeech 2019, pp. 2350–2354

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.