Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Sounding that Object: Interactive Object-Aware Image to Audio Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a diffusion model's learned text-image attention can be replaced at test time by a user-selected segmentation mask to generate the selected object's sound, and supports this with a formal error bound and experiments.

desk verdict Solid empirical result on interactive object-to-audio generation; the theory section oversells a conditional bound as a validation. read the letter →

arxiv 2506.04214 v1 pith:LXOY7CXC submitted 2025-06-04 cs.CV cs.LGcs.MMcs.SDeess.AS

Tingle Li , Baihe Huang , Xiaobin Zhuang , Dongya Jia , Jiawei Chen , Yuping Wang , Zhuo Chen , Gopala Anumanchipalli
show 1 more author
Yuxuan Wang
This is my paper
classification cs.CVcs.LGcs.MMcs.SDeess.AS
keywords interactiveaudiogenerationobject-awareimage-to-audiosegmentation-maskconditioningattention-maskequivalencelatentdiffusionmodelcontrastiveaudio-visualgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a user can select an object in an image, such as a car or a dog, and have a generative audio model synthesize that object's sound, even in a scene full of competing sound sources. The central claim is that the model's learned text-image attention weights are functionally equivalent to test-time segmentation masks, so swapping in a mask from a segmentation model keeps the audio aligned with the chosen object. The paper proves a test-time error bound for this substitution and reports state-of-the-art sound-object alignment, with the method outperforming prior image-to-audio and text-to-audio baselines. If true, this gives creative tools and assistive systems a simple click-to-sound interface that avoids the omissions and sound bindings that plague prompt-based audio generation in multi-source scenes.

What carries the argument

The key machinery is multi-modal scaled dot-product attention between text embeddings (queries) and image-patch embeddings (keys/values), feeding a latent diffusion audio model. The attention weights are treated as a learned soft mask, and at test time they are replaced by a normalized SAM segmentation mask of a user-selected object. The theoretical keystone is Theorem 3.1, which bounds the expected test-time error in terms of the segmentation error, the contrastive loss of the encoders, and Lipschitz constants, showing the substitution is controlled.

What would settle it

A concrete test is to compute, on held-out images with annotated object masks and per-object sound labels, the average overlap (for example, intersection over union) between the model's text-image attention map and the ground-truth mask; if attention and mask are nearly uncorrelated, or if swapping the attention for the mask changes human-rated object-relevance substantially, the functional-equivalence claim is undermined. A second direct test is to measure the model's InfoNCE loss and compare it with the optimal value, since Theorem 3.1 relies on that gap being small.

Watch

Extended reading notes

Core claim

The paper discovers that a conditional latent diffusion model trained with scaled dot-product attention between CLAP text embeddings and CLIP image patch embeddings learns attention weights that approximate the posterior probability that each image patch generates the described sound. Because the softmax over patch attention matches the form of the InfoNCE contrastive loss, the trained attention map acts like a soft segmentation mask, and the model remains well-conditioned when those attention weights are replaced by a hard SAM segmentation mask at test time. This is the property that enables interactive, object-level audio generation from a single image without ever training on mask labels.

Load-bearing premise

The theoretical guarantee assumes that the fine-tuned attention weights stay close to the contrastive optimum of the frozen CLIP and CLAP encoders, so that the contrastive error is small, but the paper never measures that error or confirms it for its trained model.

Editorial extensions

If this is right

  • A single image can drive a click-to-audio workflow in which each mouse-selected object yields its own soundtrack.
  • The method outperforms text- and image-conditioned baselines on objective sound-event accuracy and human-rated relevance-to-object scores.
  • The model transfers to out-of-domain scenes and datasets without retraining, as shown on VGG-Sound, ImageHear, and Places-derived images.
  • Stronger segmentation modules yield better audio, consistent with the error bound that treats mask error as a driver of generation error.
  • Because the diffusion model still sees the whole image, multi-object selections blend into a single coherent soundscape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same attention-as-mask principle may extend to other generation tasks where a contrastively pretrained text-image encoder can localize a region of interest, such as object-conditioned image editing or region-conditioned video generation.
  • The empirical equality of attention and mask performance holds even though the assumed small contrastive loss is never directly measured, so the substitution's practical viability does not depend on the theoretical guarantee being airtight.
  • Because the model learns soft, sound-relevant attention rather than hard full-object masks, its masks are naturally concentrated on sound-emitting parts such as a dog's head, a property that could be exploited to improve grounding in future models.
  • The interaction paradigm could be extended by using negative masks to suppress unwanted sounds, yielding a mask-based audio editing interface not described in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an interactive object-aware audio generation system. A latent diffusion model is fine-tuned so that text-image dot-product attention (with CLIP image features, CLAP text features, and learnable projections) grounds generation in relevant image regions. At test time, the attention weights are replaced by SAM segmentation masks, so that a user can click on an object and obtain audio matching that object. The manuscript claims a theoretical validation (Theorem 3.1) that attention functionally approximates segmentation masks, and presents quantitative comparisons, ablations, cross-dataset evaluations, and human studies on AudioCaps, VGG-Sound, and ImageHear.

Significance. If the empirical claims hold, this is a practically useful and well-evaluated contribution: the idea of substituting segmentation masks for attention at test time is simple, and the ablations (especially mask-at-training vs mask-at-inference) are informative. The paper ships broad comparisons, a human study with controls, and a self-contained proof in the appendix. However, the headline theoretical validation is not established: Theorem 3.1 is a perturbation bound whose right-hand side contains error terms that are neither measured nor minimized by the training objective, and the paper's own empirical observations in Section 4.3 suggest those terms may not be small. The practical substitution is supported by the empirical ablation, not by the theorem as stated.

major comments (3)
  1. [Section 3.3, Eq. (6), Theorem 3.1] The theorem only bounds test error in terms of unmeasured quantities, in particular epsilon_contrast = E_q D_KL(p_q || u_q) and epsilon_sam = E_q ||m_q - p_q||_1. The model is trained with the denoising objective L_theta (Eq. 1); nothing in that objective minimizes epsilon_contrast, and the attention weights u_q rely on fine-tuned projections W^Q, W^K, W^V rather than the frozen CLIP/CLAP embeddings. CLIP/CLAP pretraining optimizes global image-text matching, not patch-level sound-source posteriors p_q = P(t_q | i_{q,k}), so the assertion that 'massive training' has minimized these errors is unsupported. To make the claimed validation load-bearing, the authors should measure these quantities on a held-out set (e.g., with human-annotated sound-source masks as a proxy for p_q) or explicitly reframe the theorem as a conditional guarantee that does not by itself justify the substitution.
  2. [Section 4.3, Section 4.4] The paper's own analysis contradicts the smallness of epsilon_sam. Section 4.3 states that 'masking entire object regions imposes an overly rigid prior, as sound is typically emitted from specific parts (e.g., a dog's head rather than its tail)' and that soft attention 'directly approximates the ground-truth distribution,' while Section 4.4 describes SAM masks as 'hard attention' covering entire objects. If the ground-truth mask p_q is a sound-source posterior, then a whole-object SAM mask is not close to p_q for many objects, so the theorem's epsilon_sam term need not be small. This tension should be resolved either by redefining p_q as an object-level mask (which would make the 'sound-relevant subregion' observation moot) or by quantifying attention-mask disagreement and its effect on the bound.
  3. [Section 3.3, Eq. (5)] The claimed equivalence between InfoNCE and the MLE loss of softmax attention weights is not established. The InfoNCE loss in Eq. (5) is defined over global text-image pairs, whereas the attention weights are computed over image patches; the manuscript does not specify the 'proper scaling' that makes the two losses coincide, and the substitution of patches for independent samples changes the optimization target. In addition, the encoder placement in Eq. (5) appears to be swapped: it writes <E_v(x_T), E_t(x_I_1)>, which evaluates the image encoder on text and the text encoder on an image. This makes the theoretical motivation in Section 3.3 difficult to follow and should be corrected or the equivalence should be stated as a loose analogy rather than an exact reduction.
minor comments (6)
  1. [Section 4.2, Table 1] The text says the method 'outperforms across metrics,' but Table 1 shows that SSV2A has slightly better FAD (1.265 vs 1.271) and AVC (0.893 vs 0.891). While the differences are small and ACC and REO are clearly better, the blanket claim should be qualified.
  2. [Section 3.3, Eq. (5)] As noted in the major comments, the roles of E_t and E_v in Eq. (5) appear reversed relative to the definitions of text and image encoders; please fix the notation.
  3. [Table 3] The header 'Muiti-Head Attn.' has a typo; it should read 'Multi-Head Attn.'.
  4. [Section 3.3, Theorem 3.1] The Lipschitz assumptions on f, f*, and v are not stated with respect to which arguments; for example, v takes three arguments (s, i, m), and the bound uses changes in both the first and third arguments. Please specify the relevant norms and which variables are held fixed.
  5. [Appendix C] The human evaluation description says each participant spent at least 90 seconds on each sample; with 100 samples and 50 participants this would require a very large total annotation time. Clarify whether 'each sample' means each HIT or whether the 90-second threshold applies to the entire batch.
  6. [Section 4.3] The phrase 'hard masks sampled from the ground-truth distribution exhibit high variance' is unclear because SAM masks are deterministic outputs, not samples from a distribution; please clarify the intended probabilistic statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 3.1 is a conditional error bound; the unverified smallness of its error terms is a validity concern, not a circular reduction.

full rationale

The paper's central theoretical claim, Theorem 3.1, is a valid error decomposition rather than a circular construction. The test-time gap between the optimal model using the ground-truth mask p_q and the deployed model using the SAM mask m_q is bounded through triangle inequalities and Lemma F.1 by error terms ϵV, ϵsam, ϵcontrast, and ϵf. Lemma F.1 itself is Pinsker's inequality applied to the definitional identity ϵcontrast = E_q[D_KL(p_q || u_q)], which follows from the definitions of ϵcontrast and u_q; this is a conditional statement, not the paper's conclusion. The paper's assertion that these error terms 'have been minimized by massive training' is unsupported because CLIP/CLAP training optimizes global image-text matching rather than patch-level sound-source posteriors, and Section 4.3 concedes that SAM masks highlight entire objects while attention focuses on sound-relevant subregions, so ϵsam is not guaranteed small. This is a validity gap in the claimed guarantee, not a circular reduction by construction. The empirical core of the paper—the ablation comparing attention to masks, the quantitative gains over SSV2A, and the human REO evaluation—is measured independently of the theorem. Self-citations are peripheral and not load-bearing for the main derivation. Therefore no prediction or first-principles result reduces to its input by definition or by self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on pretrained CLIP/CLAP encoders, the AudioLDM backbone, and SAM, all taken from prior work. The paper's own contribution adds fine-tuned projections and an attention-to-mask substitution whose theoretical support depends on unmeasured small-error assumptions. The only constructed object is an unobserved ground-truth mask distribution p_q, and no new physical or model entities are introduced.

free parameters (2)
  • CFG guidance scale λ = 2.0
    Headline tables use λ=2.0, chosen after sweeping λ=1.0 to 3.0 on the test set (Appendix D Table 4); guidance scale changes ACC, FAD, KL, IS and AVC materially.
  • Audio-visual matching threshold = 0.6
    Dataset filtering threshold chosen by scanning 0.4 to 0.8 and selecting the best test metrics (Appendix D Table 5); this changes the training data and all reported numbers.
assumptions (4)
  • ad hoc to paper InfoNCE loss for CLAP/CLIP, after substituting image patches for samples, equals the MLE loss of the softmax attention weights under proper scaling.
    The equivalence is asserted in Section 3.3 with 'under proper scaling' but never derived; Eq. 5 appears to swap E_v and E_t arguments, so the bridge between contrastive learning and attention is not cleanly established.
  • ad hoc to paper The fine-tuned attention module inherits the contrastive optimality of the frozen CLIP/CLAP encoders, so ϵ_contrast is small at test time.
    Lemma F.1 and Theorem 3.1 require the model's attention weights to be close to p_q; the paper does not measure ϵ_contrast for the trained projections, and training uses only the audio reconstruction loss, not mask labels.
  • domain assumption A ground-truth patch-level mask p_q = P(t_q|i_q,k) exists and SAM masks approximate it with small ℓ1 error.
    p_q is defined in Section 3.3 but never observed; Figure 4 gives qualitative attention-mask similarity only, with no quantitative ℓ1 comparison, so small ϵ_sam is an assumption.
  • standard math The composed network f and the value function v satisfy the stated Lipschitz conditions with finite constants.
    The paper cites standard Lipschitz bound references but does not compute constants, so the bound is qualitative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sounding that Object: Interactive Object-Aware Image to Audio Generation." pith.science (2026). https://pith.science/paper/LXOY7CXC

@misc{pith2026250604214,
  author       = {Pith},
  title        = {Pith review of: Sounding that Object: Interactive Object-Aware Image to Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LXOY7CXC}},
  note         = {Machine review of arXiv:2506.04214}
}
read the original abstract

Generating accurate sounds for complex audio-visual scenes is challenging, especially in the presence of multiple objects and sound sources. In this paper, we propose an {\em interactive object-aware audio generation} model that grounds sound generation in user-selected visual objects within images. Our method integrates object-centric learning into a conditional latent diffusion model, which learns to associate image regions with their corresponding sounds through multi-modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the {\em object} level. We theoretically validate that our attention mechanism functionally approximates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving better alignment between objects and their associated sounds. Project page: https://tinglok.netlify.app/files/avobject/

Figures

Figures reproduced from arXiv: 2506.04214 by the authors.

Figure 1
Figure 1. Interactive object-aware audio generation. We generate sound aligned with specific visual objects in complex scenes. Users can select one or more objects in the scene using segmentation masks, and our model generates audio corresponding to the selected objects. Here, we show a busy street with multiple sound sources (left). After training, our model generates object-specific audio (right), such as crowd noise for pe… view at source ↗
Figure 2
Figure 2. Model architecture. We encode the reference spectrogram via a pre-trained latent encoder. An image and text prompt are processed by separate encoders, and their embeddings are fused using an attention mechanism to highlight relevant objects. We then feed these conditioned features and noisy latent into a latent diffusion model to generate the object-specific audio. Finally, the latent decoder reconstructs the spectr… view at source ↗
Figure 3
Figure 3. Qualitative model comparison. We show audio generation results for our method and the baselines, each of which is conditioned on an image, text, or segmentation mask. 4.2. Comparison to Baselines Quantitative results [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visualization results. We visualize the difference between attention maps and segmentation masks using im￾ages from Places (Zhou et al., 2017) and text prompts from BLIP (Li et al., 2022a). could be because addition operations are not compatible with the contrastive lo…
Figure 5
Figure 5. Figure 5: Interactive audio generation. Our model gener￾ates object-specific sounds in the city (left) and beach (right) scenes, and composes a complete soundscape when one or more objects are selected. Input Image Mask Output Audio Input Image Mask Output Audio [PITH_FULL_IMAG…
Figure 6
Figure 6. Figure 6: Generating soundscapes from visual texture changes.. We generate different soundscapes by manipulat￾ing the visual textures of the same scene, such as changing weather (left) or materials (right). directly highlighting entire selected objects, our model gen￾erates soft…
Figure 7
Figure 7. Figure 7: Distribution of matching scores. We present the scores for audio-visual pairs in the AudioSet. ViTs BEATs Adaptor Adaptor Transformer Classifier Input Audio Input Video [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Model architecture of audio-visual matching. We train a model to quantify the correspondence between a video and its corresponding soundtrack. Caption rephrasing. To ensure captions to focus exclu￾sively on visible sounding objects, we utilize Llama (Tou￾vron et al., 2…
Figure 9
Figure 9. Figure 9: Prompt for Llama. We extract common features between the audio and visual caption using Llama, ensuring the resulting caption focuses on events present in both modalities while avoiding overly specific details. Vehicle 31.6% Animal 19.7% Environment 15.8% Human 6.3% 4.…
Figure 10
Figure 10. Figure 10: Categorical distribution of the filtered Au￾dioSet. We show top 8 categories derived from AudioSet annotations. B.2. Dataset Configuration [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Human evaluation interface. We show the interface used for the subjective evaluation of generated audio samples. Participants are presented with input text, an image, and a corresponding audio sample, and are instructed to rate the audio on four criteria. All ratings …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages

  1. [1]

    S., and Zisserman, A

    Afouras, T., Owens, A., Chung, J. S., and Zisserman, A. Self-supervised learning of audio-visual objects from video. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part XVIII 16, pp. 208–224. Springer,

    work page 2020

  2. [5]

    Effect of positional encoding.We assess the impact of positional encoding (PE) on our model’s performance

    We empirically find that the model achieves optimal performance at a threshold of 0.6. Effect of positional encoding.We assess the impact of positional encoding (PE) on our model’s performance. As shown in Table 6, removing positional encoding leads to a significant degradation across all metrics, highlighting its importance in the model’s overall perform...

    work page 2024

  3. [7]

    L., Wu, H.-H., Salamon, J., and Bello, J

    Cramer, A. L., Wu, H.-H., Salamon, J., and Bello, J. P. Look, listen, and learn more: Design choices for deep audio embeddings. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852–3856. IEEE,

    2019

  4. [8]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding.arXiv preprint arXiv:1810.04805,

    10 Sounding that Object: Interactive Object-Aware Image to Audio Generation Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding.arXiv preprint arXiv:1810.04805,

    arXiv

  5. [9]

    Clipsep: Learning text-queried sound separation with noisy unlabeled videos.arXiv preprint arXiv:2212.07065,

    Dong, H.-W., Takahashi, N., Mitsufuji, Y ., McAuley, J., and Berg-Kirkpatrick, T. Clipsep: Learning text-queried sound separation with noisy unlabeled videos.arXiv preprint arXiv:2212.07065,

    arXiv

  6. [11]

    We randomly selected 100 samples for evaluation, each rated by 50 unique participants to ensure reliability

    Each participant was presented with an input image and the corresponding generated audio, then rated each sample on a scale from 1 to 5 based on the following criteria: (i) Overall Quality (OVL), assess- ing the general audio quality; (ii) Relevance to Input Text (RET), measuring the alignment of the audio with the asso- ciated text description; (iii) Rel...

    work page 2012

  7. [12]

    B., and Tor- ralba, A

    Gan, C., Huang, D., Chen, P., Tenenbaum, J. B., and Tor- ralba, A. Foley music: Learning to generate music from videos. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part XI 16, pp. 758–775. Springer,

    work page 2020

  8. [13]

    Gotta hear them all: Sound source aware vision to audio generation.arXiv preprint arXiv:2411.15447,

    Guo, W., Wang, H., Ma, J., and Cai, W. Gotta hear them all: Sound source aware vision to audio generation.arXiv preprint arXiv:2411.15447,

    arXiv

Show all 41 references
  1. [14]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  2. [16]

    D., Kim, B., Lee, H., and Kim, G

    Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Paper...

    2019

  3. [19]

    Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis.Advances in Neural Information Processing Systems, 33:17022–17033, 2020a

    Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis.Advances in Neural Information Processing Systems, 33:17022–17033, 2020a. Kong, Q., Xu, Y ., Iqbal, T., Cao, Y ., Wang, W., and Plumb- ley, M. D. Acousti...

    2019

  4. [20]

    H., Kim, S., Yoo, I., Yang, F., Cho, D., Kim, Y ., Chang, H., Kim, J., and Kim, S

    Lee, S. H., Kim, S., Yoo, I., Yang, F., Cho, D., Kim, Y ., Chang, H., Kim, J., and Kim, S. Soundini: Sound- guided diffusion for natural video editing.arXiv preprint arXiv:2304.06818,

    arXiv

  5. [21]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization.arXiv preprint arXiv:1711.05101,

    arXiv

  6. [23]

    Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    arXiv

  7. [24]

    A., Zhang, R., and Zhu, J.-Y

    Park, T., Efros, A. A., Zhang, R., and Zhu, J.-Y . Con- trastive learning for unpaired image-to-image translation. InComputer Vision–ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pp. 319–345. Springer,

    2020

  8. [25]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714,

    Ravi, N., Gabeur, V ., Hu, Y .-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714,

    arXiv

  9. [26]

    Self-supervised audio-visual co- segmentation

    Rouditchenko, A., Zhao, H., Gan, C., McDermott, J., and Torralba, A. Self-supervised audio-visual co- segmentation. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2357–2361. IEEE,

    2019

  10. [27]

    and Adi, Y

    Sheffer, R. and Adi, Y . I hear your true colors: Image guided audio generation. InICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  11. [28]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502,

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502,

    2010 arXiv

  12. [29]

    Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  13. [30]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418,

    V oita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418,

    1905 arXiv

  14. [31]

    P., and Salamon, J

    Wu, H.-H., Nieto, O., Bello, J. P., and Salamon, J. Audio- text models do not yet leverage natural language. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  15. [32]

    Auffusion: Leveraging the power of diffusion and large language models for text- to-audio generation.arXiv preprint arXiv:2401.01044,

    Xue, J., Deng, Y ., Gao, Y ., and Li, Y . Auffusion: Leveraging the power of diffusion and large language models for text- to-audio generation.arXiv preprint arXiv:2401.01044,

    arXiv

  16. [33]

    D., et al

    Yuan, Y ., Jia, D., Zhuang, X., Chen, Y ., Liu, Z., Chen, Z., Wang, Y ., Wang, Y ., Liu, X., Plumbley, M. D., et al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.04416,

    arXiv

  17. [34]

    Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds.arXiv preprint arXiv:2407.01494,

    Zhang, Y ., Gu, Y ., Zeng, Y ., Xing, Z., Wang, Y ., Wu, Z., and Chen, K. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds.arXiv preprint arXiv:2407.01494,

    arXiv

  18. [35]

    Results Video We provide a results video on the project webpage, which showcases our model’s ability to generate sounds based on the masked object prompts

    15 Sounding that Object: Interactive Object-Aware Image to Audio Generation A. Results Video We provide a results video on the project webpage, which showcases our model’s ability to generate sounds based on the masked object prompts. Specifically, this video demon- strates th...

    2017

  19. [36]

    The original dataset comprises 4,616 hours of video clips, each paired with corresponding labels and captions

    as the primary source for this task. The original dataset comprises 4,616 hours of video clips, each paired with corresponding labels and captions. Inspired by Sound-VECaps (Yuan et al., 2024), we apply the following refinement steps to adapt the dataset for our use. Audio-vis...

    2024

  20. [37]

    Speech" and “Music

    annotations. We uniformly sample 48 hours across these categories for the test set, with the remaining used for training. Notably, there is no overlap between training and testing videos. As most clips contain multiple sound sources, we randomly select 100 examples from the te...

    2017

  21. [40]

    By extracting features from both modalities and computing cosine similarity, we show in Table 9 that our method consistently outperforms baselines on this metric

    to measure audio-visual matching on static images. By extracting features from both modalities and computing cosine similarity, we show in Table 9 that our method consistently outperforms baselines on this metric. E. Additional Dataset Evaluations VGG-Sound dataset.To further ...

    2020

  22. [41]

    video clips with better audio-visual synchronization, for test- ing

    0.39 Ours0.45 Table 9: Comparison of ImageBind (IB) scores across dif- ferent methods. video clips with better audio-visual synchronization, for test- ing. To obtain captions aligned with this dataset, we apply the same refinement procedure described in Appendix B.1. We assess...

    2023

  23. [1994]

    P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A

    Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation.arXiv preprint arXiv:1901.11390,

    1901 arXiv

  24. [2015]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

    arXiv

  25. [2016]

    D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J

    Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Stable audio open. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2025

  26. [2017]

    Diff-foley: Syn- chronized video-to-audio synthesis with latent diffusion models.arXiv preprint arXiv:2306.17203,

    Luo, S., Yan, C., Hu, C., and Zhao, H. Diff-foley: Syn- chronized video-to-audio synthesis with latent diffusion models.arXiv preprint arXiv:2306.17203,

    arXiv

  27. [2018]

    Neural machine translation by jointly learning to align and translate.arXiv preprint arXiv:1409.0473,

    Bahdanau, D. Neural machine translation by jointly learning to align and translate.arXiv preprint arXiv:1409.0473,

    arXiv

  28. [2019]

    Visual acoustic matching

    Chen, C., Gao, R., Calamia, P., and Grauman, K. Visual acoustic matching. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18858–18868, 2022a. Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual datas...

    2020

  29. [2020]

    Audio-visual synchronisation in the wild.arXiv preprint arXiv:2112.04432, 2021a

    Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., and Zisserman, A. Audio-visual synchronisation in the wild.arXiv preprint arXiv:2112.04432, 2021a. Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., and Zisserman, A. Localizing visual sounds the hard way. InPro...

    arXiv

  30. [2021]

    Synch- former: Efficient synchronization from sparse cues

    Iashin, V ., Xie, W., Rahtu, E., and Zisserman, A. Synch- former: Efficient synchronization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5325–5329. IEEE,

    2024

  31. [2022]

    On uni-modal feature learning in supervised multi-modal learning

    Du, C., Teng, J., Li, T., Liu, Y ., Yuan, T., Wang, Y ., Yuan, Y ., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning. InInternational Conference on Machine Learning, pp. 8632–8656. PMLR, 2023a. Du, Y ., Chen, Z., Salamon, J., Russell, B., and Owens...

    2023

  32. [2023]

    S., Wiles, O., Moses, Y ., and Zisserman, A

    Koepke, A. S., Wiles, O., Moses, Y ., and Zisserman, A. Sight to sound: An end-to-end approach for visual piano transcription. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1838–1842. IEEE,

    2020

  33. [2024]

    K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A., and Mitsufuji, Y

    Cheng, H. K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A., and Mitsufuji, Y . Taming multimodal joint training for high-quality video-to-audio synthesis.arXiv preprint arXiv:2412.15322, 2024a. Cheng, X., Zheng, S., Wang, Z., Fang, M., Zhang, Z., Huang, R., Ma, Z., Ji, S....

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.