Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A contrastive model trained on mixed audio naturally locks onto one dominant sound source, and the paper turns that bias into a two-stage method that also finds the quieter source.

desk verdict A useful two-stage method and a needed evaluation fix, held back by protocol-mixing in the main table and an unmeasured identification step at the core. read the letter →

arxiv 2608.05816 v1 pith:V3U2RLTZ submitted 2026-08-06 cs.MM

Han Hu , Dongheng Lin , Yuqi Hou , Haotian Li , Hyung Jin Chang , Jianbo Jiao This is my paper · ORCID
classification cs.MM
keywords audio-visuallocalisationself-supervisedlearningcontrastiveselectiveconvergencedual-sourcesoundspatialpriorsegmentation-maskevaluationsourceseparation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the circular dependency of multi-source audio-visual localisation—separating audio requires knowing where sources are, localising requires separated audio—can be broken by an emergent bias of contrastive learning. When a contrastive model is trained on mixed audio, it naturally locks onto the single most salient sound-source correspondence, a behaviour the authors call selective convergence. The paper builds a two-stage framework that first harvests that bias into a dominant-source prior and then uses the prior to unmask the remaining source, achieving the best numbers among self-supervised methods on dual-source benchmarks without any manual labels. It also argues that the standard bounding-box evaluation is systematically biased and contributes a large pixel-level mask benchmark, VGGSound-DuetMask, so that localisation quality is measured against actual object boundaries. If true, the practical payoff is a scalable, annotation-free route to multi-source localisation that treats a known failure mode of contrastive learning as a useful spatial prior.

What carries the argument

Selective convergence is the paper's name for the tendency of contrastive audio-visual training on mixed audio to concentrate the learned similarity map on a single, most salient source correspondence instead of spreading across all sources. Stage 1 obtains a heatmap from the dot product $\langle V_i, a_i \rangle$ and a contrastive loss with differentiable thresholding; this heatmap is post-processed into a clean spatial prior $M_{\mathrm{dom}}$. Stage 2 partitions visual features as $V_1 = V \odot M_{\mathrm{dom}}$ and $V_2 = V \odot (1 - M_{\mathrm{dom}})$, then a cross-attention audio decoupler uses each visual partition as keys and values to extract source-specific audio features, and source-specific contrastive losses produce the final maps. The supporting theorem shows that the gradient update strengthens only the dominant audio component $c_1$ while leaving $c_2$ unchanged, so any initial gap $g_1 > g_2$ is amplified into winner-take-all selection.

What would settle it

Build a synthetic dual-source test set in which one object is small and visually inconspicuous but clearly audible, while a large, motion-rich, silent object dominates the frame. Train only Stage 1 with the paper's setup; if the heatmap concentrates on the silent object, or if Stage 2 then fails to recover the true subdominant source, the claim that selective convergence supplies a reliable spatial prior fails. At the theory level, measuring the initial per-region similarity gap $g_1-g_2$ across random initialisations would decide whether that gap is a pre-existing premise or an artefact of training.

Watch

Extended reading notes

Core claim

The paper's central claim is that contrastive audio-visual training on mixed audio does not attempt to represent all sound sources equally: it selectively converges to the most salient audio-visual correspondence, and this bias can be exploited rather than suppressed. The paper formalises this as SCAV, a two-stage framework in which Stage 1 learns a heatmap for the dominant source and Stage 2 uses that heatmap as a spatial prior to partition visual features and cross-attend the mixed audio, yielding two independent localisation maps. The paper further claims that this approach outperforms all self-supervised methods on dual-source benchmarks and even surpasses some weakly supervised methods on certain metrics, and that bounding-box evaluation systematically overrewards coarse activation, motivating a new segmentation-mask benchmark.

Load-bearing premise

The pipeline assumes that whatever object Stage 1 selectively converges to is actually producing the dominant sound; because the paper defines dominance as whatever the model picks, a wrong pick corrupts the spatial prior and can cause Stage 2 to discard the true subdominant source.

Editorial extensions

If this is right

  • Because selective convergence is a property of the training objective rather than a hand-crafted rule, any contrastive audio-visual model trained on mixed audio can supply the dominant-source prior for free, with no annotations.
  • Progressive two-stage training outperforms joint end-to-end training: an unstable early prior misleads the decoupler, so letting Stage 1 converge first is a necessary condition for Stage 2's gains.
  • The source-wise protocol used by earlier benchmarks hides cross-source false positives, so published IoU numbers for some multi-source methods are inflated; the derivation in the paper shows that source-wise IoU always upper-bounds frame-wise IoU for the same predictions.
  • Box-based metrics reward activating background pixels inside the rectangle, whereas SCAV's gains are larger under the new segmentation-mask evaluation, indicating that the improvement is genuine localisation rather than coarse rectangle-filling.
  • The performance gap over self-supervised baselines widens on the larger and more diverse VGGSound-Duet set, suggesting the benefit of leveraging selectivity grows with data scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is iterating the same select-then-unmask recipe beyond two sources: after Stage 2 produces two maps, a third stage could mask both learned sources and re-run the decoupler.
  • Selective convergence may be a general explanation for why many contrastive audio-visual models fail on multi-source scenes; prior methods that engineer against this bias with hand-tuned thresholds may be fighting the learning dynamics instead of using them.
  • Stage 2's cross-attention audio decoupler effectively performs audio separation from a visual prior, so the same architecture could be turned into a separation model by reading out the decoupled audio features, a testable extension.
  • Using SAM-generated masks as training pseudo-labels rather than only as a test benchmark could further remove the need for bounding boxes, but that goes beyond what the paper claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SCAV, a fully self-supervised two-stage framework for dual-source audio-visual localisation built on an observed phenomenon the authors call selective convergence: contrastive audio-visual models trained on mixed audio naturally focus on one dominant source rather than representing all sources equally. Stage 1 trains encoders with a contrastive loss and differentiable thresholding to produce a dominant-source map M_dom; Stage 2 uses M_dom as a spatial prior to partition visual features, decouples audio and visual features via cross-attention, and trains source-specific contrastive losses to localise the subdominant source. The paper also argues that the established bounding-box evaluation protocol inflates scores, introduces a new pixel-level mask benchmark, VGGSound-DuetMask, and reports state-of-the-art results among self-supervised methods. A theoretical analysis of selective convergence is provided in Appendix E, and a reproduction of the NoPrior baseline under a frame-wise protocol is documented in Appendix F.

Significance. If the selective-convergence claim holds, it is a genuinely interesting observation that reframes a known limitation of contrastive learning as a usable inductive bias, and the two-stage framework is a plausible way to break the circular dependency of multi-source localisation. The paper also makes two useful community contributions: a formal analysis of source-wise versus frame-wise evaluation protocols (Appendix D, Eq. (24)-(25)) and a new large-scale mask-annotated benchmark, VGGSound-DuetMask. The authors are commendably explicit about the scope of their theoretical result, stating in Appendix E that their analysis amplifies an assumed gap g1 > g2 rather than predicting which source is selected, and they provide a detailed reproduction of a baseline in Appendix F. However, the load-bearing empirical claim—that selective convergence selects a true sound source—is not directly measured, and the headline comparison in Table 1 mixes evaluation protocols, so the current evidence does not fully support the reported performance claims.

major comments (4)
  1. [Sec. 4.3, Table 1, Appendix D] The headline comparison in Table 1 mixes evaluation protocols. The table note states that Mix-and-Localize is re-run under the paper's frame-wise evaluation for VGGSound-Duet, while AVGN and OA-SSL are taken as originally reported. By the paper's own account in Appendix D, the original VGGSound-Duet evaluation is source-wise: activations on the opposite half-frame are discarded before IoU is computed. Eq. (25) shows that source-wise IoU is always at least frame-wise IoU for the same predictions. Therefore SCAV's frame-wise numbers are not directly comparable to AVGN and OA-SSL's original source-wise numbers, and the claims of 'best performance among self-supervised methods' and 'surpasses some weakly-supervised approaches' are not supported by Table 1 as presented. Please re-run AVGN and OA-SSL under the same frame-wise protocol (as was done for NoPrior), or alternatively report SCAV under the source-wise protocol, so that all entries in Table 1 are apples-to-apples.
  2. [Sec. 4.5, Eq. (18), Table 4; Appendix E] The paper does not directly measure whether the Stage 1 model selects a true sound-producing source. In Sec. 4.5, the dominant source is defined post hoc as the ground-truth mask with the higher IoU (Eq. (18)); samples where both IoUs are near zero are absorbed into the 'dominant source' statistics, so Table 4 cannot distinguish true selection from a model that simply biases toward one arbitrary region. Appendix E is explicit that its analysis assumes an initial gap g1 > g2 and proves amplification, not selection of a true source. Since Sec. 3.2 defines dominance as 'whichever source the model converges to,' the central premise that M_dom is a usable spatial prior remains unverified. Please add a direct selection-accuracy measurement, e.g., the fraction of test samples where M_dom's peak or binarised region has IoU above a meaningful threshold with either ground-truth mask, and report the false-selection rate where both IoUs are near zero. Such a measurement would close the gap between the observed selective-convergence phenomenon and the claim that it identifies a true sound source.
  3. [Sec. 3.3, Eq. (9), Appendix A.2] Stage 2's hard partition V2 = V ⊙ (1 - M_dom) assumes that M_dom does not suppress the true subdominant source. The paper's own failure analysis in Appendix A.2 documents false activations on visually salient but non-sounding regions, e.g., a performer's mouth instead of the instrument. If such a false region becomes M_dom, the true subdominant source is either discarded if it overlaps with the false region, or it is forced into the same branch as the remaining foreground and background; there is no self-supervised signal in Stage 2 that recovers a source lost at the partition. The failure-case discussion acknowledges this qualitatively, but no quantitative analysis is provided. Please add an ablation that corrupts M_dom with controlled false regions or noise and measures Stage 2's sensitivity, so the reader can assess how often false selection propagates and whether the framework is robust to imperfect priors.
  4. [Appendix E, Eq. (3), Lemma 2] The theoretical analysis rests on conditions that are not verified for the actual training setup. Lemma 2 requires the threshold to satisfy ar{S}_1 - δ > ε_p > ar{S}_2 + δ, but the thresholds ε_p = 0.65 and ε_n = 0.4 in Eq. (3) are fixed while the similarity values S(p) depend on the learned feature scales. Assumptions 1-3 also postulate orthonormal audio source features and very low visual noise, which are strong idealisations. As stated, Theorem 1 characterises an idealised amplification mechanism and does not establish that the Stage 1 training operates in this regime. Please either verify the threshold condition empirically (e.g., by inspecting the distributions of S(p) on source and background regions during training) or clarify in the main text that the theory is an illustrative analysis rather than a proof that selective convergence occurs under the exact training conditions.
minor comments (5)
  1. [Sec. 4.2, Eq. (15)] The definition of CAP uses AP_k, which is never defined; please clarify whether it is the average precision for class k and how the binary decision is made for each source pair.
  2. [Table 2] Please specify explicitly whether all methods on VGGSound-DuetMask were re-run by the authors under the same frame-wise, mask-based protocol, or whether some numbers are taken from original papers. The current table caption does not state this, and the protocol is critical for interpreting the comparison.
  3. [Appendix F] The reproduction changes the concatenation dimension in datasets_flow.py from dim=1 to dim=2 to obtain a horizontal layout. Since convolutional and positional-encoding layers may be sensitive to the spatial arrangement, please verify that the original model was intended for horizontal concatenation or discuss whether the dimension change could affect the reproduced results.
  4. [Algorithm 1] In the definition of the background embedding vneg, the weight N(i,j) is multiplied by S(i,j); please state explicitly whether S(i,j) is the raw similarity score or some normalised version, since the notation is not defined at that point.
  5. [Throughout] The method is referred to as both 'Mix-and-Localize' and 'Mix-and-Localise' in different places; please standardise the spelling.

Circularity Check

1 steps flagged · score 3.0 of 10

Appendix E's explanation of selective convergence labels the eventual winner as the larger-initial-gap source, so the theoretical 'derivation' is conditional on the phenomenon it claims to explain; the central benchmark results remain externally validated.

  1. self definitional [Appendix E.1-E.4 (Problem Setup, Proposition 1, Theorem 1); Sec. 3.1-3.2 definition of dominant source]
    "without loss of generality, we label the leading region — the one the model ends up converging to — as source 1, so that g1 > g2. This labels the eventual winner rather than predicting it: our analysis characterises how an existing gap is amplified, not what creates it. ... the gap S1 − S2 ≈ g1 − g2 > 0 follows directly from the labeling g1 > g2."

    The Appendix claims to theoretically justify selective convergence, but the initial imbalance g1 > g2 is not established independently: it is assigned by labeling source 1 as 'the one the model ends up converging to.' Proposition 1 then derives the regional similarity gap 'directly from the labeling,' and Theorem 1 concludes that gradient updates increase c1, i.e. the model converges to source 1. The conclusion is therefore the same fact used to define the assumption. The paper explicitly acknowledges that it 'labels the eventual winner rather than predicting it,' so the analysis is conditional on the very outcome it is meant to explain.

full rationale

The paper's core empirical contributions — the observed concentration of the first-stage heatmap on one source (Sec. 4.5), the two-stage framework, and the quantitative results on MUSIC-Duet, VGGSound-Instruments, and VGGSound-Duet — are checked against external datasets and ground-truth masks, so the reported numbers are not self-referential. No parameter is fitted to a benchmark and then renamed as a prediction, and no load-bearing claim rests on a self-citation: reference [16] is a related-work citation only, and the simplicity-bias connection in Appendix E.5 is attributed to the external work of Xue et al. [38]. The one identifiable circular step is in Appendix E: the formal 'explanation' of selective convergence labels the eventual winner as the source with larger initial similarity g1 > g2, derives the similarity gap from that labeling, and then proves that the same source wins. The paper is transparent about this scope, stating that it 'labels the eventual winner rather than predicting it,' so this does not corrupt the empirical evaluation. However, as a first-principles derivation of the phenomenon it reduces to an assumption of the outcome. That is a partial, localized circularity, not a collapse of the paper's central claims, hence a score of 3 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces the conceptual label 'selective convergence' and the benchmark VGGSound-DuetMask, but neither is a postulated physical entity. The theoretical analysis relies on idealized assumptions about feature orthogonality and low noise, which are recorded above as axioms and should be verified empirically.

free parameters (3)
  • positive threshold epsilon_p = 0.65
    Used in Eq. 3 to define positive spatial masks; taken from prior contrastive localization work [6], hand-chosen and directly shapes Stage 1 selectivity.
  • negative threshold epsilon_n = 0.4
    Used in Eq. 3 to define negative spatial masks; hand-chosen from prior work [6] and affects which regions are suppressed.
  • sigmoid temperature tau = 0.03
    Used in Eq. 3 to control mask sharpness; hand-chosen from prior work [6] and critical to the winner-take-all concentration exploited by the method.
assumptions (4)
  • domain assumption Mixed audio decomposes as an additive mixture amix = a1 + a2 in the embedded audio feature space.
    Used in Sec. 3.1 and Appendix E.1; exact for waveforms but not guaranteed after nonlinear audio feature extraction.
  • domain assumption The two visual sources occupy disjoint spatial regions and the Stage 1 mask M_dom is a valid spatial prior for one true source.
    Used in Sec. 3.3 Eq. 9 to partition visual features; synthetic benchmarks enforce disjoint halves, but real scenes need not, and false activations are acknowledged in Appendix A.2.
  • ad hoc to paper Audio sources are orthogonal in feature space and visual features are low-noise, as stated in Assumptions 1-3 of Appendix E.1.
    These idealized assumptions make the selective-convergence theorem tractable, but they are not verified on real learned features.
  • domain assumption Differentiable thresholding with fixed epsilon and tau is a faithful approximation to the training objective used in the proof.
    Eq. 3; inherited from prior work, but the theoretical concentration result uses tau tending to 0 while training uses tau = 0.03.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence." pith.science (2026). https://pith.science/paper/V3U2RLTZ

@misc{pith2026260805816,
  author       = {Pith},
  title        = {Pith review of: Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3U2RLTZ}},
  note         = {Machine review of arXiv:2608.05816}
}
read the original abstract

Localising multiple sound sources in visual scenes remains a fundamental challenge in multimodal perception due to an inherent circular dependency: separating mixed audio requires knowing source locations, while identifying sound-producing regions requires separated audio signals. In this paper, we focus on the dual-source setting and discover a selective convergence in self-supervised audio-visual learning: when presented with multiple sound sources, contrastive models naturally converge to the most salient audio-visual correspondence rather than attempting to represent all sources equally. This emergent phenomenon, analogous to human selective auditory attention, enables us to break the above circular dependency through a progressive two-stage framework: first, leveraging selective convergence to identify dominant sources, and then exploiting these learned priors to uncover remaining sources. Our self-supervised approach achieves the best performance among self-supervised methods on dual-source benchmarks without requiring any manual annotations, and even surpasses some weakly-supervised approaches \red{on certain metrics. Furthermore, we identify a fundamental evaluation inconsistency in existing benchmarks: comparing continuous localisation heatmaps against bounding-box annotations creates systematic biases, particularly for non-axis-aligned objects where the bounding box includes substantial background regions. To address this, we introduce pixel-level segmentation masks to the existing benchmark, enabling spatially-aligned evaluation. Together, these results suggest that embracing rather than suppressing selectivity offers a scalable, annotation-free route to multi-source localisation.

Figures

Figures reproduced from arXiv: 2608.05816 by the authors.

Figure 1
Figure 1. Top-left [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of our Selective Convergence Audio-Visual localisation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Bounding box evaluation can be misleading. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of multi-source sound localisation. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Examples of selective convergence behaviour of the first stage. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 36 canonical work pages

  1. [1]

    In: Proceedings of the IEEE international conference on computer vision

    Arandjelovic, R., Zisserman, A.: Look, listen and learn. In: Proceedings of the IEEE international conference on computer vision. pp. 609–617 (2017)

    work page 2017

  2. [2]

    In: Proceedings of the Euro- pean conference on computer vision (ECCV)

    Arandjelovic, R., Zisserman, A.: Objects that sound. In: Proceedings of the Euro- pean conference on computer vision (ECCV). pp. 435–451 (2018)

    work page 2018

  3. [3]

    Advances in neural information processing systems29(2016)

    Aytar, Y., Vondrick, C., Torralba, A.: Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems29(2016)

    work page 2016

  4. [4]

    Frontiers in Neuroscience11, 159 (2017)

    Brattico, P., Brattico, E., Vuust, P.: Global sensory qualities and aesthetic experi- ence in music. Frontiers in Neuroscience11, 159 (2017)

    work page 2017

  5. [5]

    Attention, Perception, & Psychophysics77(5), 1465– 1487 (2015)

    Bronkhorst, A.W.: The cocktail-party problem revisited: early processing and se- lection of multi-talker speech. Attention, Perception, & Psychophysics77(5), 1465– 1487 (2015)

    work page 2015

  6. [6]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., Zisserman, A.: Localiz- ing visual sounds the hard way. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16867–16876 (2021)

    work page 2021

  7. [7]

    In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio- visual dataset. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 721–725. IEEE (2020)

    work page 2020

  8. [8]

    In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Choe, J., Oh, S.J., Lee, S., Chun, S., Akata, Z., Shim, H.: Evaluating weakly supervised object localization methods right. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    work page 2020

Show all 41 references
  1. [9]

    In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV) (2025)

    Choi, H., Lee, J., Kwak, N.: What’s making that sound right now? video-centric audio-visual localization. In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV) (2025)

    2025

  2. [10]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)

    Gao, S., Chen, Z., Chen, G., Wang, W., Lu, T.: Avsegformer: Audio-visual seg- mentation with transformer. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). vol. 38, pp. 12155–12163 (2024)

    2024

  3. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Guo, R., Ying, X., Chen, Y., Niu, D., Li, G., Qu, L., Qi, Y., Zhou, J., Xing, B., Yue, W., Shi, J., Wang, Q., Zhang, P., Liang, B.: Audio-visual instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  4. [12]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Guo, Y., Ma, S., Su, H., Wang, Z., Zhao, Y., Zou, W., Sun, S., Zheng, Y.: Dual mean-teacher: An unbiased semi-supervised framework for audio-visual source lo- calization. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 36, pp. 1–14 (2023)

    2023

  5. [13]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    2016

  6. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hu, D., Nie, F., Li, X.: Deep multimodal clustering for unsupervised audiovisual learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9248–9257 (2019)

    2019

  7. [15]

    Advances in Neural Information Processing Systems33, 10077–10087 (2020)

    Hu, D., Qian, R., Jiang, M., Tan, X., Wen, S., Ding, E., Lin, W., Dou, D.: Dis- criminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems33, 10077–10087 (2020)

    2020

  8. [16]

    In: British Machine Vision Conference (BMVC) (2025)

    Hu, H., Lin, D., Huang, Q., Hou, Y., Chang, H.J., Jiao, J.: Audio-visual separation with hierarchical fusion and representation alignment. In: British Machine Vision Conference (BMVC) (2025)

    2025

  9. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (2022) Whence the Voice? 33

    Hu, X., Chen, Z., Owens, A.: Mix and localize: Localizing sound sources in mix- tures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (2022) Whence the Voice? 33

    2022

  10. [18]

    In: British Machine Vision Conference (BMVC) (2025)

    Juanola, X., Morais, G., Fuentes, M., Haro, G.: Learning from silence and noise for visual sound source localization. In: British Machine Vision Conference (BMVC) (2025)

    2025

  11. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kim, D., Um, S.J., Lee, S., Kim, J.U.: Learning to visually localize sound sources from mixtures without prior source knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26467–26476 (2024)

    2024

  12. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Kim, J., Lee, J., Seo, S.e., Yang, J., Kim, J.U.: Improving sound source localization with joint slot attention on image and audio. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 26477– 26486 (2025)

    2025

  13. [21]

    arXiv preprint arXiv:1412.6980 (2014)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  14. [22]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

    2023

  15. [23]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Liu, X., Qian, R., Zhou, H., Hu, D., Lin, W., Liu, Z., Zhou, B., Zhou, X.: Visual sound localization in the wild by cross-modal interference erasing. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1801–1809 (2022)

    2022

  16. [24]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Mahmud, T., Li, Y., Tian, Y.: T-vsl: Text-guided visual sound source localization in mixtures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 23456–23465 (2024)

    2024

  17. [25]

    In: European Confer- ence on Computer Vision

    Mo, S., Morgado, P.: Localizing visual sounds the easy way. In: European Confer- ence on Computer Vision. pp. 218–234. Springer (2022)

    2022

  18. [26]

    Mo, S., Tian, Y.: Audio-visual grouping network for sound localization from mix- tures.In:ProceedingsoftheIEEE/CVFconferenceoncomputervisionandpattern recognition. pp. 10565–10574 (2023)

    2023

  19. [27]

    arXiv preprint arXiv:2409.00486 (2024)

    Mo, S., Wang, H.: Multi-scale multi-instance visual sound localization and segmen- tation. arXiv preprint arXiv:2409.00486 (2024)

    2024 arXiv

  20. [28]

    arXiv preprint arXiv:1807.03748 (2018)

    van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  21. [29]

    In: Proceedings of the European conference on computer vision (ECCV)

    Owens, A., Efros, A.A.: Audio-visual scene analysis with self-supervised multi- sensory features. In: Proceedings of the European conference on computer vision (ECCV). pp. 631–648 (2018)

    2018

  22. [30]

    In: European conference on computer vi- sion

    Owens, A., Wu, J., McDermott, J.H., Freeman, W.T., Torralba, A.: Ambient sound provides supervision for visual learning. In: European conference on computer vi- sion. pp. 801–816. Springer (2016)

    2016

  23. [31]

    Park, S., Senocak, A., Chung, J.S.: Can clip help sound source localization? In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 5699–5708 (2024)

    2024

  24. [32]

    In: European Conference on Computer Vision

    Qian, R., Hu, D., Dinkel, H., Wu, M., Xu, N., Lin, W.: Multiple sound sources localization from coarse to fine. In: European Conference on Computer Vision. pp. 292–308. Springer (2020)

    2020

  25. [33]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Senocak, A., Oh, T.H., Kim, J., Yang, M.H., Kweon, I.S.: Learning to localize sound source in visual scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4358–4366 (2018)

    2018

  26. [34]

    IEEE Transactions on Pattern Analysis and Machine Intelligence42(8), 1984–1997 (2020)

    Senocak, A., Oh, T.H., Kim, J., Yang, M.H., Kweon, I.S.: Learning to localize sound sources in visual scenes: Analysis and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence42(8), 1984–1997 (2020)

    2020

  27. [35]

    Hu et al

    Sun, W., Zhang, J., Wang, J., Liu, Z., Zhong, Y., Feng, T., Guo, Y., Zhang, Y., Barnes, N.: Learning audio-visual source localization via false negative aware con- 34 H. Hu et al. trastive learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2023

  28. [36]

    Journal of Speech, Language, and Hearing Research60(10), 2989–3000 (2017)

    Sussman, E.S.: Auditory scene analysis: An attention perspective. Journal of Speech, Language, and Hearing Research60(10), 2989–3000 (2017)

    2017

  29. [37]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Um, S.J., Kim, D., Lee, S., Kim, J.U.: Object-aware sound source localization via audio-visual scene understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8342–8351 (2025)

    2025

  30. [38]

    In: International Conference on Machine Learning (2023)

    Xue, Y., Joshi, S., Gan, E., Chen, P.Y., Mirzasoleiman, B.: Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. In: International Conference on Machine Learning (2023)

    2023

  31. [39]

    Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., Torralba, A.: Thesoundofpixels.In:ProceedingsoftheEuropeanconferenceoncomputervision (ECCV). pp. 570–586 (2018)

    2018

  32. [40]

    arXiv preprint arXiv:2301.13190 (2023)

    Zhou, J., Shen, X., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., Zhong, Y.: Audio-visual segmentation with semantics. arXiv preprint arXiv:2301.13190 (2023)

    2023 arXiv

  33. [41]

    In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

    Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., Zhong, Y.: Audio-visual segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.