Object-aware Sound Source Localization via Audio-Visual Scene Understanding

Dongjin Kim; Jung Uk Kim; Sangmin Lee; Sung Jin Um

REVIEW 4 major objections 4 minor 49 references

Object-aware Sound Source Localization via Audio-Visual Scene Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that sound source localization becomes fine-grained when a multimodal language model labels, during training, which visible objects are making sound and which are silent, and two new losses use those labels to align and…

desk verdict Novel MLLM-anchored training recipe with consistent gains, but caption quality is unmeasured and the ORI loss is under-specified; deserves a conditional accept after revision. read the letter →

arxiv 2506.18557 v2 pith:2IYFNMJE submitted 2025-06-23 cs.CV

Sung Jin Um , Dongjin Kim , Sangmin Lee , Jung Uk Kim This is my paper · ORCID

classification cs.CV

keywords soundsourcelocalizationaudio-visualcorrespondencemultimodallargelanguagemodelscontrastivelearningobjectregionisolationMUSICdatasetVGGSoundmulti-source

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing sound source localization fails in cluttered scenes because it matches audio to object categories rather than object states: a guitar hanging on a wall and a guitar being played produce the same category match. The proposed fix is to use a multimodal large language model during training to caption each frame into sound-making foreground objects and silent background objects, then use those captions as reference anchors for the visual and audio features. Two new losses, the Object-aware Contrastive Alignment loss and the Object Region Isolation loss, pull the features of the true sound source toward its caption embedding, push silent objects away, and force the heatmaps of different sound sources not to overlap. The authors report that this reaches the top of both single-source and multi-source localization benchmarks on MUSIC and VGGSound, with particularly large gains on the duet test sets. If true, the approach shows that state-level scene understanding, not just category-level correspondence, is the missing ingredient in audio-visual localization.

What carries the argument

The load-bearing object is a caption-conditioned reference feature set produced by an MLLM (InternVL 2.0-8B in the main runs) during training only: for each frame the MLLM outputs $K$ foreground captions and one background caption, given the audio class label. A text encoder (BERT, using the [CLS] token embedding) turns those into foreground reference features $\mathbf{F}_r^p$ and a background reference feature $\mathbf{l}_r^n$. The Object-aware Contrastive Alignment loss then uses these as anchors in a contrastive form: foreground visual region features are attracted to the averaged foreground reference, background visual region features to the background reference, with a similarity threshold $\tau$ to remove false negatives. The Object Region Isolation loss flattens per-reference similarity maps and uses Sinkhorn-regularized optimal transport to compute first-order Wasserstein distances between every pair of maps, pushing them apart so multiple sound sources do not share one heatmap. The whole training objective is $\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{oca} + \lambda_2 \mathcal{L}_{ori}$, with $\lambda_1=1$ and $\lambda_2=0.1$.

What would settle it

Take a fixed set of VGGSound or MUSIC frames, run the captioning pipeline, and score the generated foreground and background captions against human annotations or ground-truth object-action labels (e.g., whether the described object is actually the one emitting the audio). Then measure localization performance on only the clips where captions are fully correct. If the reported gains disappear on the correct-caption subset relative to the noisy-caption subset, the improvement would be shown to come from something other than scene understanding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an audio-visual model can be taught to localize not just the category that matches a sound, such as 'guitar', but the specific object that is producing it, such as 'the guitar being played', and that the teaching signal can come from MLLM-generated captions. The method yields a foreground reference feature from text embeddings of $K$ sound-making captions and a background reference feature from a silent-object caption, then uses two losses. The Object-aware Contrastive Alignment loss pulls the visual feature of the sound-associated region toward the averaged foreground reference while pushing the background-region feature away and discarding near-duplicate negatives; the Object Region Isolation loss compares similarity maps of each source and background reference and minimizes their pairwise first-order Wasserstein distance, encouraging spatially separated heatmaps. The reported outcome is the top of both single- and multi-source tables: with ResNet18, CAP of 61.4 on MUSIC-Duet and 45.9 on VGGSound-Duet, against 52.1 and 32.5 for the previous best method NoPrior; with AudioCLIP, CAP of 64.1 and 47.1, ahead of the previous AudioCLIP-based method T-VSL at 62.9 and 35.7.

Load-bearing premise

The load-bearing premise is that the MLLM-generated foreground and background captions are accurate enough to serve as training targets; if a caption names the wrong object as the sound source or invents a silent object, the two loss functions will reinforce that error.

Editorial extensions

If this is right

If the central claim is right, localization models no longer need to match audio to object categories; they can be steered by state-level captions, so scenes with visually identical playing and non-playing instruments become separable.
The gains on MUSIC-Duet and VGGSound-Duet imply that multi-source localization can be improved substantially without changing the encoder architecture, only by adding reference-driven alignment and isolation losses.
Because the MLLM is discarded at inference, the method keeps the inference-time cost of a standard two-stream encoder; the supplementary comparison reports faster inference and lower memory than the previous text-guided baseline.
The VGGSound-Trio result suggests the framework handles a flexible number of sound sources rather than a fixed $K$.
The supplementary experiments across three MLLMs and two text encoders imply the benefit comes from the caption format itself, not from one specific language model or text encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the same caption-conditioned alignment could transfer directly to audio-visual segmentation, where every pixel needs the same foreground versus background distinction, not just a coarse heatmap.
Editorial extension: because caption generation happens once at training start, the method could be folded into a bootstrap loop in which the model's own confident localizations rewrite the captions for a second training pass.
Testable extension: injecting controlled caption noise, such as swapping foreground and background labels on a fraction of frames, should degrade CAP monotonically; that would directly confirm the captions, not the losses, carry the reported gain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the paper has a genuinely new training recipe for audio-visual sound source localization—using an MLLM to generate separate foreground and background captions from the image plus the audio class label, then using those as anchors for two contrastive/optimal-transport losses. The reported gains are consistent across MUSIC and VGGSound, single and multi-source, and two backbones, and the ablations show both losses contribute. The code and generated captions are public. That's real value.

The soft spots are real but not disqualifying. The biggest is exactly what the stress-test note flags: the MLLM never sees the audio, only the class label, so the foreground/background captions can be wrong when the label is ambiguous or the scene has multiple instances. The paper shows qualitative examples but no accuracy metric for captions, and no ablation that replaces MLLM captions with ground-truth annotations. That's a genuine hole because the whole method rests on caption quality. I'd want a caption accuracy measurement, even a small human-evaluated set, and ideally a robustness analysis to caption noise.

Second, the ORI loss is under-specified in the main text. Eq. 7 doesn't say how the 2D maps are normalized before Sinkhorn, and the supplementary only mentions "pixel intensity differences and Euclidean distance" in the cost matrix. That's not reproducible as written. Need a precise definition.

Third, the hyperparameters tau and lambda are tuned on the test sets (the tau ablation table uses test-set numbers), and there are no error bars anywhere. That alone would make me hesitant to trust the absolute numbers, though the pattern of improvements is probably real.

On the stress-test note specifically: it's right that the captions are not audio-grounded, but it's not fatal. The class label gives a strong prior, and visual cues (hands on instrument, motion) can disambiguate. The paper should just measure this instead of leaving it implicit.

Bottom line: solid empirical paper with a novel idea, and it deserves a serious referee. I'd conditional-accept it, asking for caption quality metrics, a precise ORI loss, and either error bars or a statement about runs. The core claim—that scene-differentiated MLLM captions help SSL—probably holds up.

Recommendation: send to peer review; don't desk reject.

Referee Report

4 major / 4 minor

Summary. This paper proposes OA-SSL, a method for audio-visual sound source localization that uses a multimodal large language model (InternVL2) at training time to generate foreground and background captions from the input image and the audio class label. The captions are encoded with BERT and used as reference anchors in two new losses: Object-aware Contrastive Alignment (OCA), which aligns visual foreground/background features with the corresponding reference features while repelling the opposite type, and Object Region Isolation (ORI), which uses a Sinkhorn-based Wasserstein distance to encourage spatial separation between distinct sound-making regions and the background. The method is evaluated on MUSIC and VGGSound for both single- and multi-source localization, reporting consistent improvements over prior methods, e.g., 9.3% CAP on MUSIC-Duet and 14.6% CAP on VGGSound-Duet with a ResNet18 backbone. The code and generated captions are announced as publicly available.

Significance. If the reported results are reliable, the paper makes a useful contribution by showing that MLLM-generated scene descriptions, even though produced without direct access to the audio, can provide training-time guidance for fine-grained visual sound-source localization. The training-only use of the MLLM keeps the inference pipeline simple and efficient, which is a practical strength. The paper also includes useful ablations and generalization studies across different MLLMs and text encoders. However, the validity of the central claim currently rests on unmeasured assumptions about the accuracy of the MLLM-generated captions and on hyperparameter selection performed on test sets. These issues, together with the absence of repeated-run statistics, mean the evidence as presented is not yet sufficient to establish the claimed state-of-the-art results.

major comments (4)

[Section 3.2, Table 5 (Supp), Eq. (4)-Eq. (7)] The MLLM receives only the image and the audio class label; it never observes the audio waveform or spectrogram, and the prompt in Table 5 instructs it to assume that the class label identifies a sound-emitting object. The generated foreground/background captions are therefore not audio-grounded, and any visually salient but silent object can be encoded as a positive reference. The paper provides no quantitative measure of caption accuracy; Supplementary Figure 2 shows only selected examples. Because OCA and ORI train against these references, systematic caption errors would be learned as correct targets and could account for part of the reported gains. Please add a quantitative evaluation of caption quality (e.g., human agreement, agreement with ground-truth sounding-object annotations, or a per-category breakdown) and an ablation that replaces MLLM captions with ground-truth or detector-based foreground/background annotations, including a corrupted-or-shuffled-caption control.
[Section 4.2, Table 4, Supp Table 3] The hyperparameters tau, lambda_1, and lambda_2 are effectively selected by direct evaluation on the test sets. Table 4 sweeps tau on both the MUSIC-Duet and VGGSound-Duet test sets, and Supplementary Table 3 sweeps (lambda_1, lambda_2) on the VGGSound-Duet test set. This constitutes test-set fitting and inflates the reported improvements. The authors should select hyperparameters on a held-out validation split (or via cross-validation) and report only the test-set numbers for the final configuration.
[Eq. (7) and Algorithm 1 (Supp)] The ORI loss applies the Sinkhorn algorithm to raw cosine-similarity maps \bar{S}^i_{rn} and 1 - \bar{S}^i_{rm}. Cosine similarities lie in [-1, 1] and the maps are not normalized to be non-negative and sum to one, while the Sinkhorn algorithm requires non-negative probability vectors. The supplementary note about incorporating pixel intensity differences and Euclidean distance into the distance matrix does not resolve the normalization issue. Please specify the exact normalization used before computing the Wasserstein distance (e.g., clipping/ReLU followed by row-wise normalization, or a softmax over spatial locations) and state the ground-cost construction actually used in the reported experiments.
[Tables 1-3, 5] All experiments report a single training run without error bars or significance tests. Given that the headline gains are 9.3% CAP on MUSIC-Duet and 14.6% CAP on VGGSound-Duet, and that training stochasticity in this setting is nontrivial, the absence of repeated-run statistics makes it hard to judge whether the differences are reliable. Please report mean and standard deviation over at least three seeds for the main comparisons and the ablation study.

minor comments (4)

[Tables 1, 4, 5 and supplementary tables] The metric header "CloU@0.3" should be "CIoU@0.3" for consistency with the text.
[Section 4.2] The text describes tau as "the weighting parameter" for the OCA loss, but in Eq. (4) tau is a similarity threshold used to identify false negatives; please correct the wording.
[Throughout] The dataset name is written inconsistently as both "VGG-Sound" and "VGGSound"; please unify the spelling.
[Supplementary Material, Section 3] The video demo paragraph still contains the string "CVPR2025 SubmissionID 698", which should be removed or replaced before publication.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the training signal comes from an external MLLM/BERT and evaluation is against external benchmarks, with only a minor non-load-bearing self-citation.

full rationale

I checked the claimed derivation chain and found no step in which a prediction reduces by construction to its inputs. The OCA and ORI reference features (Eqs. 4-7) are BERT embeddings of captions generated by InternVL 2.0-8B from the image and audio class label (Sec. 3.2 and Supp. Table 5); these captions are not the model's own outputs, are not fitted to the test metrics, and are not used to construct the final heatmap. The predicted localization map S_a is the cosine similarity between visual and audio features (Eq. 1), not the text anchors, so the inference-time prediction is not the training target by construction. The only overlapping-author citation is the adoption of NoPrior's iterative object-identification algorithm [18] in Sec. 3.1 and its use as a comparison baseline; that prior work is externally published, and the component does not encode the paper's fine-grained foreground/background distinction. The unmeasured accuracy of the MLLM captions is a genuine data-quality risk, but that would be an error-in-labels problem rather than circular reasoning. This is a self-contained empirical paper, so the circularity score is minimal.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method rests on two external knowledge sources (MLLM captions, BERT embeddings), one borrowed component (NoPrior iterative identification), and an under-specified optimal-transport loss. No new physical entities are introduced. The main free parameters were tuned on the test set.

free parameters (3)

tau (false-negative threshold) = 0.7
Chosen by sweeping on VGGSound-Duet test set (Table 4); controls which similar foreground references are treated as false negatives in OCA loss.
lambda_1 (OCA loss weight) = 1.0
Chosen alongside lambda_2 on VGGSound-Duet test set (supplementary Table 3).
lambda_2 (ORI loss weight) = 0.1
Chosen alongside lambda_1 on VGGSound-Duet test set (supplementary Table 3).

assumptions (3)

domain assumption InternVL 2.0-8B produces semantically correct foreground/background captions from image and class label
The training targets are derived from these captions; no quantitative validation is provided (Section 3.2, supplementary Figure 2).
ad hoc to paper Cosine similarity maps can be treated as probability distributions for Wasserstein distance without normalization
The ORI loss in Eq. 7 applies Sinkhorn to raw Sim values; no normalization or non-negativity handling is specified (Section 3.4, supplementary Algorithm 1).
domain assumption NoPrior's iterative object identification correctly estimates the number and locations of sound sources
The multi-source pipeline adopts the algorithm from [18] and inherits its assumptions (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Object-aware Sound Source Localization via Audio-Visual Scene Understanding." pith.science (2026). https://pith.science/paper/2IYFNMJE

@misc{pith2026250618557,
  author       = {Pith},
  title        = {Pith review of: Object-aware Sound Source Localization via Audio-Visual Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2IYFNMJE}},
  note         = {Machine review of arXiv:2506.18557}
}

read the original abstract

Audio-visual sound source localization task aims to spatially localize sound-making objects within visual scenes by integrating visual and audio cues. However, existing methods struggle with accurately localizing sound-making objects in complex scenes, particularly when visually similar silent objects coexist. This limitation arises primarily from their reliance on simple audio-visual correspondence, which does not capture fine-grained semantic differences between sound-making and silent objects. To address these challenges, we propose a novel sound source localization framework leveraging Multimodal Large Language Models (MLLMs) to generate detailed contextual information that explicitly distinguishes between sound-making foreground objects and silent background objects. To effectively integrate this detailed information, we introduce two novel loss functions: Object-aware Contrastive Alignment (OCA) loss and Object Region Isolation (ORI) loss. Extensive experimental results on MUSIC and VGGSound datasets demonstrate the effectiveness of our approach, significantly outperforming existing methods in both single-source and multi-source localization scenarios. Code and generated detailed contextual information are available at: https://github.com/VisualAIKHU/OA-SSL.

Figures

Figures reproduced from arXiv: 2506.18557 by the authors.

**Figure 2.** Network configuration of the proposed sound source localization framework. MLLM is used only during the training process and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Visualization results for VGGSound-Single test set. We [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Visualization results for (a) MUSIC-Duet and (b) VGGSound-Duet test set. We compare our method with NoPrior [ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 1.** Figure 1: Additional visualization results for VGGSound-Duet test set. [PITH_FULL_IMAGE:figures/full_fig_p012_1.png]

**Figure 2.** Figure 2: Visualization results of generated captions on VGGSound train set. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 41 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Objects that sound

Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018

work page 2018
[3]

Vggsound: A large-scale audio-visual dataset

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020

work page 2020
[4]

Localizing visual sounds the hard way

Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, 2021

work page 2021
[5]

Exploring simple siamese representation learning

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021

2021
[6]

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024

work page 2024
[7]

Sinkhorn distances: Lightspeed computation of optimal transport

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013

2013
[8]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

2009

Show all 49 references

[9]

Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks

Haoyi Duan, Yan Xia, Zhou Mingze, Li Tang, Jieming Zhu, and Zhou Zhao. Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. In NeurIPS, 2024

2024
[10]

With a little help from my friends: Nearest-neighbor contrastive learning of visual representations

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021

2021
[11]

Hear the flow: Optical flow-based self-supervised visual sound source localization

Dennis Fedorishin, Deen Dayal Mohan, Bhavin Jawade, Srirangaraj Setlur, and Venu Govindaraju. Hear the flow: Optical flow-based self-supervised visual sound source localization. In WACV, 2023

2023
[12]

Audioclip: Extending clip to image, text and audio

Andrey Guzhov, Federico Raue, J \"o rn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP, 2022

2022
[13]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

2016
[14]

Deep multimodal clustering for unsupervised audiovisual learning

Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In CVPR, 2019

2019
[15]

Discriminative sounding objects localization via self-supervised audiovisual matching

Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. In NeurIPS, 2020

2020
[16]

Mix and localize: Localizing sound sources in mixtures

Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In CVPR, 2022

2022
[17]

Boosting contrastive self-supervised learning with false negative cancellation

Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In WACV, 2022

2022
[18]

A review of recent advances on deep learning methods for audio-visual speech recognition

Denis Ivanko, Dmitry Ryumin, and Alexey Karpov. A review of recent advances on deep learning methods for audio-visual speech recognition. Mathematics, 2023

2023
[19]

Learning to visually localize sound sources from mixtures without prior source knowledge

Dongjin Kim, Sung Jin Um, Sangmin Lee, and Jung Uk Kim. Learning to visually localize sound sources from mixtures without prior source knowledge. In CVPR, 2024

2024
[20]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[21]

Unsupervised sound localization via iterative contrastive learning

Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Unsupervised sound localization via iterative contrastive learning. CVIU, 2023

2023
[22]

Exploiting transformation invariance and equivariance for self-supervised sound localisation

Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In ACM MM, 2022

2022
[23]

Generalized video anomaly event detection: Systematic taxonomy and comparison of deep models

Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng Sun, and Liang Song. Generalized video anomaly event detection: Systematic taxonomy and comparison of deep models. ACM Computing Surveys, 2024

2024
[24]

T-vsl: Text-guided visual sound source localization in mixtures

Tanvir Mahmud, Yapeng Tian, and Diana Marculescu. T-vsl: Text-guided visual sound source localization in mixtures. In CVPR, 2024

2024
[25]

Localizing visual sounds the easy way

Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In ECCV, 2022 a

2022
[26]

A closer look at weakly-supervised audio-visual source localization

Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. In NeurIPS, 2022 b

2022
[27]

Audio-visual grouping network for sound localization from mixtures

Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. In CVPR, 2023

2023
[28]

Audio-visual scene analysis with self-supervised multisensory features

Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018

2018
[29]

Multiple sound sources localization from coarse to fine

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In ECCV, 2020

2020
[30]

Multimodal open-vocabulary video classification via pre-trained vision and language models

Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui. Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint arXiv:2207.07646, 2022

2022 arXiv
[31]

Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation

Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In CVPR, 2023

2023
[32]

Learning to localize sound source in visual scenes

Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In CVPR, 2018

2018
[33]

Learning sound localization better from semantically similar samples

Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In ICASSP, 2022

2022
[34]

Sound source localization is all about cross-modal alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source localization is all about cross-modal alignment. In ICCV, 2023

2023
[35]

Unsupervised sounding object localization with bottom-up and top-down attention

Jiayin Shi and Chao Ma. Unsupervised sounding object localization with bottom-up and top-down attention. In WACV, 2022

2022
[36]

Flowgrad: Using motion for visual sound source localization

Rajsuryan Singh, Pablo Zinemanas, Xavier Serra, Juan Pablo Bello, and Magdalena Fuentes. Flowgrad: Using motion for visual sound source localization. In ICASSP, 2023

2023
[37]

Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes

Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes. In CVPR, 2022

2022
[38]

Learning audio-visual source localization via false negative aware contrastive learning

Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware contrastive learning. In CVPR, 2023

2023
[39]

Audio-visual spatial integration and recursive attention for robust sound source localization

Sung Jin Um, Dongjin Kim, and Jung Uk Kim. Audio-visual spatial integration and recursive attention for robust sound source localization. In ACM MM, 2023

2023
[40]

Watch video, catch keyword: Context-aware keyword attention for moment retrieval and highlight detection

Sung Jin Um, Dongjin Kim, Sangmin Lee, and Jung Uk Kim. Watch video, catch keyword: Context-aware keyword attention for moment retrieval and highlight detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7473--7481, 2025

2025
[41]

V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models

Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In AAAI, 2024

2024
[42]

Multimodal large language models: A survey

Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In IEEE BigData, 2023

2023
[43]

Sonicvisionlm: Playing sound with vision language models

Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Sonicvisionlm: Playing sound with vision language models. In CVPR, 2024

2024
[44]

A proposal-based paradigm for self-supervised sound source localization in videos

Hanyu Xuan, Zhiliang Wu, Jian Yang, Yan Yan, and Xavier Alameda-Pineda. A proposal-based paradigm for self-supervised sound source localization in videos. In CVPR, 2022

2022
[45]

A survey on multimodal large language models

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

2023 arXiv
[46]

Video-llama: An instruction-tuned audio-visual language model for video understanding

Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

2023 arXiv
[47]

The sound of pixels

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018

2018
[48]

Weakly supervised contrastive learning

Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In ICCV, 2021

2021
[49]

Exploiting visual context semantics for sound source localization

Xinchi Zhou, Dongzhan Zhou, Di Hu, Hang Zhou, and Wanli Ouyang. Exploiting visual context semantics for sound source localization. In WACV, 2023

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Objects that sound

Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018

work page 2018

[3] [3]

Vggsound: A large-scale audio-visual dataset

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020

work page 2020

[4] [4]

Localizing visual sounds the hard way

Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, 2021

work page 2021

[5] [5]

Exploring simple siamese representation learning

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021

2021

[6] [6]

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024

work page 2024

[7] [7]

Sinkhorn distances: Lightspeed computation of optimal transport

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013

2013

[8] [8]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

2009