Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

Gotta Hear Them All: Towards Sound Source Aware Audio Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Explicitly modeling local sound sources—detected visually and disambiguated in a cross-modal manifold—improves image-to-audio generation, the paper claims.

desk verdict A sound-source-aware audio generation system with real artifacts (VGGS3, SSMS, CMSS) and a plausible core idea, but the SOTA claim is inflated by tuning hyperparameters on the same VGG-SS test set used for the headline numbers. read the letter →

arxiv 2411.15447 v4 pith:GVXUZCUR submitted 2024-11-23 cs.MM cs.CVcs.SDeess.AS

Wei Guo , Heng Wang , Jianbo Ma , Weidong Cai This is my paper · ORCID
classification cs.MMcs.CVcs.SDeess.AS
keywords image-to-audiogenerationsoundsourcemodelingcross-modalcontrastivelearningvision-to-audiosynthesismultimodalcompositionmatchingscoreaudioVGGS3dataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that image-to-audio generation improves when the model is told about the individual sound sources in a scene rather than only the whole scene. To that end it proposes SS2A, a generator that detects objects in an image, projects their CLIP or CLAP embeddings through a contrastively learned Cross-Modal Sound Source (CMSS) manifold that separates one source from another, and then attention-mixes the source embeddings into a single CLAP audio embedding that a pretrained AudioLDM turns into a waveform. The authors claim that this source-aware conditioning surpasses existing global-scene methods on fidelity, relevance, and human opinion scores, and that it works best for scenes with several sound sources at once. A reader would care because the method points toward audio synthesis that can be controlled at the level of individual objects and combined across vision, text, and audio prompts.

What carries the argument

The load-bearing mechanism is the Cross-Modal Sound Source (CMSS) manifold: a joint semantic space, learned contrastively from single-source image-audio pairs, into which CLIP and CLAP embeddings of detected sound sources are projected so that matching visual and audio sources attract and non-matching sources repel. An auxiliary reconstructor maps CMSS embeddings back toward CLAP audio space so the manifold keeps audio character, and a Cross-Modal Contrastive Mask Regularization (CCMR) weakens the contrastive pressure on pairs that are already similar in both CLIP and CLAP space. A separate attention-based Sound Source Remixer takes the disambiguated CMSS embeddings, concatenated with their original CLIP embeddings, and mixes them into one CLAP embedding that is fed to the frozen pretrained AudioLDM generator.

What would settle it

A direct test: build a set of images in which the real sound maker is not visually detectable (a bird behind a wall, an offscreen engine) while detected objects are silent. If SS2A without a text prompt performs as well as SS2A with the prompt, or as well as a global-scene baseline, the visual source-perception step is not doing the load-bearing work; the measurable outcome is SSMS and CLIP-Score on that set.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a sound source-aware conditioning path—visual detection of objects, cross-modal translation of text or audio prompts into the same semantic space, and contrastive disambiguation of each source before mixing—lets a frozen audio diffusion model produce audio that is more relevant and more immersive than conditioning on the global image alone. The paper builds this path as SS2A and evaluates it on both single-source and multi-source image-to-audio benchmarks. It reports that SS2A achieves the best relevance and competitive fidelity in general image-to-audio tests, is superior to all compared baselines on every metric in multi-source generation, and reaches competitive video-to-audio results after attaching a temporal aggregation module. It also introduces VGGS3, a dataset of single-sound-source image-audio pairs filtered from VGGSound, and a Sound Source Matching Score (SSMS) that measures localized relevance as an F1 score over predicted sound labels.

Load-bearing premise

The load-bearing premise is that the visual detector reliably finds the objects that actually make sound: any source that is missed, too small, visually ambiguous, or silent-but-cropped will be absent from, or pollute, the conditioning, and the rest of the pipeline cannot correct for that.

Editorial extensions

If this is right

  • If SS2A is right, conditioning on detected local objects should improve audio fidelity and relevance on arbitrary images, with the largest gains in scenes that contain multiple sounding objects.
  • A contrastive manifold trained only on single-source pairs transfers to multi-source mixing, so object-level audio semantics can be learned from simpler data and composed at generation time.
  • Because all conditions are projected into one manifold, users can mix vision, text, and audio prompts to edit or steer a generated sound scene.
  • The same source-level representation, combined with a positional temporal aggregator, yields competitive video-to-audio synthesis, suggesting source modeling is not image-specific.
  • The SSMS metric gives a source-level relevance score that can be measured automatically and that the authors show can disagree with image-domain relevance scores like CLIP-Score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The detector threshold is a hidden bottleneck: if a real sound source has no visual bounding box, SS2A cannot condition on it, so the method's ceiling is tied to detection quality rather than to the manifold or remixer.
  • Because SSMS relies on an audio classifier's predicted labels, it inherits that classifier's vocabulary and blind spots; extending SSMS to another label set or open vocabulary would test whether the metric itself is stable.
  • The same architecture suggests a natural interactive editing loop: replacing or deleting one source token in the remixer's input should change only that part of the generated scene, which is testable as a controllability benchmark.
  • The reported lower sensitivity to audio prompts indicates that supplying CLIP-style semantics for audio conditions, for instance through a Wav2CLIP-style translator, is a concrete next step suggested by the paper's own limitation section.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SS2A, a sound-source-aware image-to-audio generator. SS2A first perceives sound sources in an image with a visual detector (YOLOv8x), embeds the detected crops with CLIP and optionally takes text/audio prompts via CLIP/CLAP, projects these into a learned Cross-Modal Sound Source (CMSS) manifold using a contrastive objective with a new Cross-Modal Contrastive Mask Regularization (CCMR), and then mixes the source embeddings with an attention-based Sound Source Remixer before feeding the resulting CLAP embedding into a pretrained AudioLDM generator. The paper also introduces a new single-source dataset VGGS3, a new Sound Source Matching Score (SSMS) metric, and a temporal aggregation extension for video-to-audio. The experimental section reports objective and subjective results on VGGSound, VGG-SS, MUSIC, and ImageHear, together with ablations of the CMSS manifold, CCMR, Cycle Mix, and Remixer architecture.

Significance. If the central claim holds, the paper makes a useful contribution by showing that explicitly modeling local sound sources, rather than conditioning only on a global scene, improves the relevance and multi-source fidelity of generated audio. The VGGS3 dataset and SSMS metric are potentially reusable resources, and the multimodal composition experiments demonstrate an interesting new control modality. The evidence is extensive in terms of datasets, ablations, out-of-distribution tests, and human MOS. However, the paper's headline state-of-the-art claim is currently overstated: on general VGGSound and single-source VGG-SS, a baseline (V2A-Mapper) has better FAD scores, and the key multi-source VGG-SS results are obtained after selecting several hyperparameters on the same small test set. The work is therefore a promising empirical direction whose central claim needs a corrected evaluation protocol before it can be accepted at face value.

major comments (3)
  1. [Datasets and Tables 4, 8, 9, 10] The VGG-SS test set is used both for hyperparameter selection and for reporting final results. The paper states that VGG-SS has no official train-test split and randomly samples 4.5K pairs for training and 500 pairs for testing. The ablations then select CCMR alpha (Tab. 4), Cycle Mix sample size and iterations (Tab. 8), CMSS architecture variant (Tab. 9), and Remixer attention depth (Tab. 10) by evaluating on the same 500-pair test set that is reported in Tab. 2. Since the multi-source portion of VGG-SS contains only 38 pairs, repeated selection on this set likely inflates the reported multi-source gains. Please add a proper held-out validation split (or nested cross-validation) for all hyperparameter choices, report final results on a truly unseen test set, and provide confidence intervals or significance tests for the multi-source comparisons.
  2. [Abstract, Conclusion, and Tables 1-2] The claim that SS2A 'achieves state-of-the-art performance in extensive image-to-audio tasks' is broader than the data support. In Tab. 1, V2A-Mapper has better V-FAD (0.946 vs. 1.150) and C-FAD (5.516 vs. 6.716) on the general VGGSound test set. In Tab. 2, for single-source VGG-SS, V2A-Mapper also has better V-FAD (1.666 vs. 2.815) and C-FAD (13.583 vs. 15.150). The multi-source VGG-SS and MUSIC results are favorable to SS2A, and the relevance metrics and MOS favor SS2A, but the 'state-of-the-art in all metrics' language should be restricted to the multi-source setting, and the single-source fidelity trade-off should be discussed explicitly.
  3. [Matching Score] The SSMS metric is introduced as a contribution, but its key parameter N (the number of predicted labels) is fixed to 10 without sensitivity analysis, and its validity is argued mainly through the observation that it places Oracle below SS2A on CS. Please report SSMS for several values of N, compare SSMS against the human relevance MOS on the same set of samples, and analyze whether the BEATs label vocabulary covers the VGG-SS and MUSIC source categories. Without this, the metric's role as a clear measure of localized audio relevance is not fully established.
minor comments (6)
  1. [Equation (7)] The mask definition uses alpha twice, once as a coefficient and once as an exponent; the behavior is therefore hard to interpret. Consider renaming one of the two occurrences and adding a short explanation of how the mask scales with the CLIP and CLAP similarities.
  2. [Table 3] The header 'CMSS CLIP' is ambiguous; the caption should state explicitly which rows correspond to removing the CMSS embeddings and which to removing the CLIP embeddings.
  3. [Tables 8-10] The column header 'MS' is used for the metric that is called 'SSMS' in the main text and other tables; please use a consistent abbreviation throughout.
  4. [Temporal Aggregation, Eq. (11)-(12)] The text says 1024 is the positional embedding's frequency resolution, while 1024 appears as the base in the denominator of the positional encoding; please clarify the notation so that the reader understands the difference between the base and the resolution.
  5. [Abstract and Demo URL] The demo website URL uses 'SSV2A' while the method is named 'SS2A'; please align the naming to avoid confusion.
  6. [Ethical Statement] The paper says the curated dataset VGGS3 will be released, but no release link or availability statement is given; please include one in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sound-source modeling claim is independently benchmarked; the VGG-SS hyperparameter selection is an evaluation-validity caveat, not a circular derivation.

full rationale

The derivation chain is self-contained rather than circular. The CMSS manifold is learned from VGGS3/VGG-SS single-source pairs with the explicit contrastive and reconstruction objectives in Eqs. (2)-(5); the Sound Source Remixer is trained on VGGSound with the objective in Eq. (9); and generation is decoded by a frozen pretrained AudioLDM. The claimed benefit of explicit sound-source modeling is tested against external baselines on VGGSound, VGG-SS, MUSIC, and ImageHear, and the key controlled ablation (Tab. 3) uses a fixed configuration across conditions rather than a fitted prediction, showing large gains from adding CMSS and CLIP semantics. The SSMS metric is a measurement tool built on the pretrained BEATs audio classifier; it is not a training objective, so it cannot force the reported improvements. Self-citations to V2A-Mapper and FRIEREN are used as baselines and building blocks, not as load-bearing justifications or uniqueness theorems. One genuine methodological caveat exists but is not circularity: the paper selects hyperparameters (CCMR alpha, CMSS architecture, Remixer depth, Cycle Mix counts) on the 500-pair VGG-SS test set, as stated in the dataset setup and Tables 4, 8, 9, and 10, which weakens the independence of those specific VGG-SS numbers. However, the central qualitative claim also survives a fixed-configuration ablation and out-of-distribution tests, so the paper's derivation does not reduce to its own inputs.

Assumptions & free parameters 8 free parameters · 5 assumptions · 1 invented entities

The central claim depends on a stack of pretrained models and learned representations. The main free parameters are hyperparameters selected on the VGG-SS test set, and the CMSS manifold is the paper's principal new latent construct with only internal evidence. The axioms reflect assumptions about detection coverage, single-source supervision, and the sufficiency of CLAP space for audio synthesis.

free parameters (8)
  • CCMR decay alpha = 0.35
    Controls mask curvature in Eq. (7); selected by ablation on VGG-SS test set (Table 4) and used throughout.
  • Cycle Mix sample size = 64
    Number of Remixer samples per iteration; tuned on VGG-SS test set (Table 8).
  • Cycle Mix iterations = 64
    Iterations of source-semantic recycling; tuned on VGG-SS test set (Table 8).
  • SSMS label count N = 10
    Number of sound source labels predicted by BEATs for the proposed metric; set by the authors and used throughout.
  • YOLOv8 confidence threshold = 0.25
    Detection threshold for perceiving visual sound sources (Supplementary Section 2.1); affects which sources enter the pipeline.
  • CaR segmentation confidence threshold = 0.5
    Used to filter single-source visual-audio pairs from VGGSound when curating VGGS3 (Supplementary Section 2.2).
  • Classifier-free guidance dropout = 0.2
    Dropout rate for zeroing tokens during Remixer training (Supplementary Section 2.1).
  • KL regularization weights lambda_1 and lambda_2 = not reported
    Weights for the KL terms in Eqs. (5) and (9); the paper does not state their values, so reimplementation would need to choose them.
assumptions (5)
  • domain assumption Pretrained foundation models (CLIP, CLAP, AudioLDM, DALL-E-2 Prior, YOLOv8, BEATs) provide faithful semantic embeddings and generation quality.
    Used throughout without verification in this paper; e.g., AudioLDM is assumed to generate high-fidelity audio from CLAP embeddings.
  • domain assumption Visual object detection is sufficient to identify sound sources in an image.
    Section 'Sound Source Perception' applies YOLOv8x at 0.25 confidence and crops detected regions; if sources are not visually detectable, the pipeline misses them.
  • domain assumption Single-source visual-audio pairs from VGG-SS, VGGS3, and translated LAION-630K pairs are representative enough to learn sound source disambiguation that generalizes to multi-source open-domain scenes.
    CMSS manifold training relies on these pairs; the paper filters noisy pairs with a Mean-Teacher and Kneedle, but the coverage assumption remains.
  • domain assumption A CLAP embedding captures the audio characteristics of a sound source, and mixing in this space is sufficient for AudioLDM to synthesize a coherent soundscape.
    The Remixer outputs a single CLAP embedding amix; all generation quality depends on this representation space.
  • domain assumption BEATs audio classifier labels are a valid proxy for the sound sources present in audio.
    The proposed SSMS metric relies on this assumption; if BEATs labels are noisy or biased, SSMS comparisons may mislead.
invented entities (1)
  • Cross-Modal Sound Source (CMSS) Manifold
    purpose: Joint semantic space where visual, text, and audio sound-source embeddings are disambiguated before mixing.
    A learned latent space introduced by the paper. Internal ablations and t-SNE analyses support its effect, but no external falsifiable prediction outside this work is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gotta Hear Them All: Towards Sound Source Aware Audio Generation." pith.science (2026). https://pith.science/paper/GVXUZCUR

@misc{pith2026241115447,
  author       = {Pith},
  title        = {Pith review of: Gotta Hear Them All: Towards Sound Source Aware Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GVXUZCUR}},
  note         = {Machine review of arXiv:2411.15447}
}
read the original abstract

Audio synthesis has broad applications in multimedia. Recent advancements have made it possible to generate relevant audios from inputs describing an audio scene, such as images or texts. However, the immersiveness and expressiveness of the generation are limited. One possible problem is that existing methods solely rely on the global scene and overlook details of local sounding objects (i.e., sound sources). To address this issue, we propose a Sound Source-Aware Audio (SS2A) generator. SS2A is able to locally perceive multimodal sound sources from a scene with visual detection and cross-modality translation. It then contrastively learns a Cross-Modal Sound Source (CMSS) Manifold to semantically disambiguate each source. Finally, we attentively mix their CMSS semantics into a rich audio representation, from which a pretrained audio generator outputs the sound. To model the CMSS manifold, we curate a novel single-sound-source visual-audio dataset VGGS3 from VGGSound. We also design a Sound Source Matching Score to clearly measure localized audio relevance. With the effectiveness of explicit sound source modeling, SS2A achieves state-of-the-art performance in extensive image-to-audio tasks. We also qualitatively demonstrate SS2A's ability to achieve intuitive synthesis control by compositing vision, text, and audio conditions. Furthermore, we show that our sound source modeling can achieve competitive video-to-audio performance with a straightforward temporal aggregation mechanism.

Figures

Figures reproduced from arXiv: 2411.15447 by the authors.

Figure 1
Figure 1. Our SS2A perceives multimodal sound sources in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of SS2A. We perceive sound sources prompted by vision, text, or audio and disambiguate them in the semantically learned CMSS Manifold, which are then mixed to generate an audio clip with immersiveness and expressiveness. mix them into a CLAP audio embedding with rich sound source information. This representation is passed to a pre￾trained audio generator, AudioLDM (Liu et al. 2023), to syn￾thesize the outpu… view at source ↗
Figure 3
Figure 3. Detailed Schematics of SS2A Modules. (a) We learn two projectors to map the CLIP-CLAP embeddings of single￾source visual-audio pairs to a joint semantic space with contrastive guidance, forming our CMSS manifold. An auxiliary CLAP reconstruction encodes audio semantics into this manifold. (b) The Sound Source Remixer attends to the CMSS embeddings concatenated with their CLIP semantics, generating a single CLAP audi… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Multimodal Sound Source Composition scenar￾ios. Our method can flexibly composite sound sources across visual, text, and audio modalities to guide V2A generation. Effect of CMSS Manifold. SS2A could learn to perform the V2A task without CMSS disambiguation. In order to…
Figure 5
Figure 5. Figure 5: Architecture of key module components. We show a single instance instead of batch inference in (b). weights at each batch. We further filter out curated/trans￾lated pairs regarded as extremely noisy from student train￾ing by computing the cosine similarity between each…
Figure 6
Figure 6. Figure 6: t-SNE visualizations of visual-audio modality alignment. The first figure visualizes raw CLIP-CLAP embeddings, the second depicts their remapped CMSS manifold embeddings and the third illustrates reconstructed CLAP embeddings from CMSS manifold. The circles mark visual…
Figure 7
Figure 7. Figure 7: Canonical plots of discriminant test. The red inner circle marks the 95% confidence interval and the red outer circle marks the 50% normal contour of audio samples. The blue circles denote the visual samples. Category Embedding Percent Misclassified↑ Entropy R2 ↓ -2 Lo…
Figure 8
Figure 8. Figure 8: Chord diagram of CMSS sound source similarities. Wider chords indicate higher similarities between sources. Embeddings Partition Coefficient↑ Raw CLIP-CLAP 8.473 CMSS Manifold 12.615 Reconstructed CLAP 12.800 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 10
Figure 10. Figure 10: Example of context awareness. Our composi￾tion transforms a normal talking man into a police officer on duty. Overlay SSV2A "Academic Conference" Overlapping Talkers without Reverbs Reverbed Academic Presentation [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Example of style awareness. Our composition changes a normal speech into an academic presentation with conference room reverb. with no interactions between these instruments, as shown in [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Screenshots of subjective survey. Each row of circles prompts a single-choice question to the testee. 9 Ethical Statement Our human evaluation is strictly anonymized without col￾lecting any sensitive personal data. We also obtain explicit verbal consent from participa…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

    cs.CV 2024-12 conditional novelty 7.0 of 10

    AV-Link unifies video-to-audio and audio-to-video generation by aligning frozen diffusion-model activations with temporally matched rotary position embeddings in a shared Fusion Block.

  2. Sounding that Object: Interactive Object-Aware Image to Audio Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A latent diffusion audio model is trained to ground sound in image patches, then uses SAM segmentation masks at test time so users can generate audio for selected objects in a scene.

  3. AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation

    cs.SD 2025-05 conditional novelty 6.0 of 10

    A training-free multi-agent framework that decomposes multimodal inputs into audio events, selects specialized generators, and self-corrects outputs to produce multiple audio types.

  4. AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

    cs.MM 2025-01 conditional novelty 6.0 of 10

    AGAV-Rater, an LMM fine-tuned in two stages, achieves state-of-the-art quality scores for AI-generated audio-visual content, text-to-audio, and text-to-music.

  5. Sound Scene Synthesis at the DCASE 2024 Challenge

    cs.AI 2025-01 conditional novelty 4.0 of 10

    Four text-to-audio systems were evaluated against a human reference in the DCASE 2024 Task 7 challenge, with a 36% quality gap and strong but small-sample FAD-to-human correlation.

Reference graph

Works this paper leans on

69 extracted references · 56 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    MaskGIT: masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: masked generative image transformer. In CVPR, pages 11315--11325, 2022

    work page 2022

  4. [4]

    VGGSound : A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound : A large-scale audio-visual dataset. In ICASSP, pages 721--725, 2020 a

    work page 2020

  5. [5]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, pages 16867--16876, 2021

    work page 2021

  6. [6]

    Visually indicated sound generation by perceptually optimized classification

    Kan Chen, Chuanxi Zhang, Chen Fang, Zhaowen Wang, Trung Bui, and Ram Nevatia. Visually indicated sound generation by perceptually optimized classification. In ECCV Workshop, 2018

    work page 2018

  7. [7]

    Deep cross-modal audio-visual generation

    Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu. Deep cross-modal audio-visual generation. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 349--357, 2017

    work page 2017

  8. [8]

    Generating visually aligned sound from videos

    Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 29: 0 8292--8302, 2020 b

    work page 2020

Show all 69 references
  1. [9]

    BEATs: audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: audio pre-training with acoustic tokenizers. In ICML, pages 5178--5193, 2023

    2023

  2. [10]

    The digital Foley : what Foley artists say about using audio synthesis

    Balandino Di Donato and Iain McGregor. The digital Foley : what Foley artists say about using audio synthesis. In Audio Engineering Society Conference: AES 2024 International Audio for Games Conference. Audio Engineering Society, 2024

  3. [11]

    CLIPSonic: text-to-audio synthesis with unlabeled videos and pretrained language-vision models

    Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serr \`a , Taylor Berg-Kirkpatrick, and Julian McAuley. CLIPSonic: text-to-audio synthesis with unlabeled videos and pretrained language-vision models. In IEEE Workshop on Applications of Signal ...

    2023

  4. [12]

    CLAP learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP learning audio concepts from natural language supervision. In ICASSP, pages 1--5, 2023

    2023

  5. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873--12883, 2021

    2021

  6. [14]

    FSD50K: an open dataset of human-labeled sound events

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 829--852, 2021

    2021

  7. [15]

    An ecological approach to auditory event perception

    William W Gaver. An ecological approach to auditory event perception. Ecological Psychology, 5 0 (1): 0 1--29, 1993

    1993

  8. [16]

    ImageBind: one embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: one embedding space to bind them all. In CVPR, pages 15180--15190, 2023

    2023

  9. [17]

    Dimensionality reduction by learning an invariant mapping

    Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, pages 1735--1742, 2006

    2006

  10. [18]

    On clustering validation techniques

    Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis. On clustering validation techniques. Journal of Intelligent Information Systems, 17: 0 107--145, 2001

    2001

  11. [19]

    CMCGAN: a uniform framework for cross-modal visual-audio mutual generation

    Wangli Hao, Zhaoxiang Zhang, and He Guan. CMCGAN: a uniform framework for cross-modal visual-audio mutual generation. In AAAI, 2018

    2018

  12. [20]

    The benefit of temporally-strong labels in audio event classification

    Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP, pages 366--370, 2021

    2021

  13. [21]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop, 2021

    2021

  14. [22]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840--6851, 2020

    2020

  15. [23]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In BMVC, 2021

    2021

  16. [24]

    Synchformer: efficient synchronization from sparse cues

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: efficient synchronization from sparse cues. In ICASSP, pages 5325--5329, 2024

    2024

  17. [25]

    Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman

    Vladimir E. Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Sparse in space and time: Audio-visual synchronisation with trainable selectors. In BMVC, 2022

    2022

  18. [26]

    Ultralytics YOLOv8 , 2023

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLOv8 , 2023. Software

    2023

  19. [27]

    Audio in vr: Effects of a soundscape and movement-triggered step sounds on presence

    Angelika C Kern and Wolfgang Ellermeier. Audio in vr: Effects of a soundscape and movement-triggered step sounds on presence. Frontiers in Robotics and AI, 7: 0 20, 2020

    2020

  20. [28]

    High-fidelity audio compression with improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN . In NeurIPS, pages 27980--27993, 2023

    2023

  21. [29]

    https://huggingface.co/nousr/conditioned-prior/tree/main/vit-l-14/aesthetic, 2024

    LAION. https://huggingface.co/nousr/conditioned-prior/tree/main/vit-l-14/aesthetic, 2024. Website

    2024

  22. [30]

    Mind the gap: understanding the modality gap in multi-modal contrastive representation learning

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, pages 17612--17625, 2022

    2022

  23. [31]

    https://github.com/haoheliu/AudioLDM, 2024

    Haohe Liu. https://github.com/haoheliu/AudioLDM, 2024. Website

    2024

  24. [32]

    AudioLDM : Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM : Text-to-audio generation with latent diffusion models. In ICML, pages 21450--21474, 2023 a

    2023

  25. [33]

    AudioLDM 2: learning holistic audio generation with self-supervised pretraining

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. AudioLDM 2: learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: ...

    2024

  26. [34]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023 b

    2023

  27. [35]

    Diff-Foley: synchronized video-to-audio synthesis with latent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-Foley: synchronized video-to-audio synthesis with latent diffusion models. In NeurIPS, pages 48855--48876, 2023

    2023

  28. [36]

    Recognition of sound sources and events

    Stephen McAdams. Recognition of sound sources and events. Thinking in sound: The cognitive psychology of human audition, pages 146--198, 1993

    1993

  29. [37]

    AVA: a large-scale database for aesthetic visual analysis

    Naila Murray, Luca Marchesotti, and Florent Perronnin. AVA: a large-scale database for aesthetic visual analysis. In CVPR, pages 2408--2415, 2012

    2012

  30. [38]

    https://huggingface.co/datasets/marianna13/BBCSoundEffects, 2024 a

    Marianna Nezhurina. https://huggingface.co/datasets/marianna13/BBCSoundEffects, 2024 a . Website

    2024

  31. [39]

    https://huggingface.co/datasets/marianna13/epidemic_sound_effects, 2024 b

    Marianna Nezhurina. https://huggingface.co/datasets/marianna13/epidemic_sound_effects, 2024 b . Website

    2024

  32. [40]

    https://github.com/openai/CLIP, 2022

    OpenAI. https://github.com/openai/CLIP, 2022. Website

    2022

  33. [41]

    https://storage.googleapis.com/openimages/web/index.html, 2024

    OpenImagesV7. https://storage.googleapis.com/openimages/web/index.html, 2024. Website

    2024

  34. [42]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In CVPR, pages 2405--2413, 2016

    2016

  35. [43]

    Masked generative video-to-audio transformers with enhanced synchronicity

    Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serr \`a . Masked generative video-to-audio transformers with enhanced synchronicity. arXiv preprint arXiv:2407.10387, 2024

    2024 arXiv

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748-...

    2021

  37. [45]

    Hierarchical text-conditional image generation with CLIP latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  38. [46]

    Fr\'echet Audio Distance: a reference-free metric for evaluating music enhancement algorithms

    Dominik Roblek, Kevin Kilgour, Matt Sharifi, and Mauricio Zuluaga. Fr\'echet Audio Distance: a reference-free metric for evaluating music enhancement algorithms. In Proc. Interspeech, pages 2350--2354, 2019

    2019

  39. [47]

    Sound synthesis and sampling

    Martin Russ. Sound synthesis and sampling. Routledge, 2012

    2012

  40. [48]

    Finding a ``kneedle" in a haystack: Detecting knee points in system behavior

    Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a ``kneedle" in a haystack: Detecting knee points in system behavior. In IEEE Int. Conf. Distr. Comput. Syst. Worksh., pages 166--171, 2011

    2011

  41. [49]

    Telecommunication Standardization Sector

    International Telecommunication Union. Telecommunication Standardization Sector. Methods for subjective determination of transmission quality. International Telecommunication Union, 1996

    1996

  42. [50]

    Audio deepfake approaches

    Ousama A Shaaban, Remzi Yildirim, and Abubaker A Alguttar. Audio deepfake approaches. IEEE Access, 11: 0 132652--132682, 2023

    2023

  43. [51]

    I hear your true colors: Image guided audio generation

    Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP, pages 1--5, 2023

    2023

  44. [52]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV, pages 3531--3539, 2021

    2021

  45. [53]

    CLIP as RNN: segment countless visual concepts without training endeavor

    Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. CLIP as RNN: segment countless visual concepts without training endeavor. In CVPR, pages 13171--13182, 2024

    2024

  46. [54]

    https://github.com/gudgud96/frechet-audio-distance, 2024

    Haohao Tan. https://github.com/gudgud96/frechet-audio-distance, 2024. Website

    2024

  47. [55]

    Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

    Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, pages 1195--1204, 2017

    2017

  48. [56]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017

    2017

  49. [57]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, page 6000–6010, 2017

    2017

  50. [58]

    V2A-Mapper: a lightweight solution for vision-to-audio generation by connecting foundation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2A-Mapper: a lightweight solution for vision-to-audio generation by connecting foundation models. In AAAI, pages 15492--15501, 2024 a

    2024

  51. [59]

    FRIEREN : Efficient video-to-audio generation with rectified flow matching

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. FRIEREN : Efficient video-to-audio generation with rectified flow matching. In NeurIPS, 2024 b

    2024

  52. [60]

    The emergence of deepfake technology: A review

    Mika Westerlund. The emergence of deepfake technology: A review. Technology innovation management review, 9 0 (11), 2019

    2019

  53. [61]

    Wav2CLIP : Learning robust audio representations from CLIP

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2CLIP : Learning robust audio representations from CLIP . In ICASSP, pages 4563--4567, 2022

    2022

  54. [62]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, pages 1--5, 2023

    2023

  55. [63]

    Seeing and Hearing: open-domain visual-audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and Hearing: open-domain visual-audio generation with diffusion latent aligners. In CVPR, pages 7151--7161, 2024

    2024

  56. [64]

    Draw an Audio: leveraging multi-instruction for video-to-audio synthesis

    Qi Yang, Binjie Mao, Zili Wang, Xing Nie, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Draw an Audio: leveraging multi-instruction for video-to-audio synthesis. arXiv preprint arXiv:2409.06135, 2024

    2024 arXiv

  57. [65]

    FoleyCrafter: bring silent videos to life with lifelike and synchronized sounds

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. FoleyCrafter: bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494, 2024

    2024 arXiv

  58. [66]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, pages 570--586, 2018

    2018

  59. [67]

    Applications of deep learning to audio generation

    Yuanjun Zhao, Xianjun Xia, and Roberto Togneri. Applications of deep learning to audio generation. IEEE Circuits and Systems Magazine, 19 0 (4): 0 19--38, 2019

    2019

  60. [68]

    Audio--visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio--visual segmentation. In ECCV, pages 386--403, 2022

    2022

  61. [69]

    Visual to sound: Generating natural sound for videos in the wild

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In CVPR, pages 3550--3558, 2018

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.