Pith. sign in

REVIEW 4 major objections 5 minor 37 references

What's Making That Sound Right Now? Video-centric Audio-Visual Localization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that temporal modeling, not just spatial alignment, is what makes audio-visual localization work in real-world video.

desk verdict A genuinely useful video-centric AVL benchmark and a plausible baseline model, but the paper's headline claim — that temporal modeling is what makes TAVLO robust — is not actually isolated by the experiments. read the letter →

arxiv 2507.04667 v2 pith:V4X6N2DN submitted 2025-07-07 cs.CV cs.AIcs.MMcs.SDeess.AS

Hahyeon Choi , Junhoo Lee , Nojun Kwak This is my paper
classification cs.CVcs.AIcs.MMcs.SDeess.AS
keywords audio-visuallocalizationvideo-centricbenchmarktemporalmodelingspatiotemporalattentionsoundsourcemultipleinstancelearningcross-eventevaluationmulti-entityscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio-visual localization is the task of saying which object in view is making the sound heard right now, and this paper argues that the field has been testing it the wrong way: mostly as a single-frame matching problem. The authors build AVATAR, a video-level benchmark with per-frame bounding-box and segmentation labels for 5,000 clips spanning four scenarios — one source, several simultaneous sources, several look-alike objects where only one sounds, and off-screen sources — plus a Cross-event subset where the active source changes over time. They also build TAVLO, a model whose audio-spatial-temporal attention binds each short audio segment to the matching frame region and tracks it across frames. On Cross-event clips, TAVLO's consensus intersection-over-union (CIoU) score drops by only 0.33 points, while image-based baselines drop by 3 to 5 points. The authors take these results as evidence that temporal modeling, not richer frame-level matching, is the decisive factor, and they offer AVATAR as a new standard for video-centric evaluation.

What carries the argument

The load-bearing component is the Audio-Spatial-Temporal (AST) Attention Block, a factorized transformer that applies multi-head self-attention first across the spatial dimension — the audio token together with all visual patch tokens at one timestamp — then across the time dimension, with a transpose between the two passes. A rectangular audio CNN kernel aligns each spectrogram segment to exactly one video frame, and spatial and temporal positional encodings keep the permutation-invariant attention aware of where and when each token is. The training objective replaces the global-audio bag used in prior multiple-instance learning with per-timestamp bags: for each frame's audio segment the positive response is the maximum similarity to any region in that frame, while negative responses are mean similarities to regions in other videos at the same timestamp. This per-frame contrastive structure is what lets the model point at the currently active emitter and ignore look-alike silent objects.

What would settle it

Take a stratified random sample of AVATAR frames, have independent annotators label the active sound sources without seeing the machine-proposed boxes, and re-run the scenario comparisons; if TAVLO's Cross-event advantage over the baselines shrinks to noise, or if inter-annotator agreement is low on Multi-entity and Mixed-sound clips, the decisive-factor claim would be in doubt.

Watch

Extended reading notes

Core claim

The central claim is that existing audio-visual localization models fail on video because they match a global audio representation to a static frame, so they cannot tell when the active sound source changes. On the new Cross-event subset, four baselines lose between 3.07 and 5.36 CIoU and up to 5.00 AUC compared with their full-benchmark scores, whereas TAVLO loses only 0.33 CIoU and 0.37 AUC. TAVLO achieves this with a factorized attention architecture that performs spatial self-attention over the audio token plus all visual tokens at each timestamp and then temporal self-attention across timestamps, trained with a frame-level multiple-instance contrastive loss in which each audio segment must match at least one region in its own frame while staying distinguishable from regions in other videos at the same time. The paper also contributes AVATAR's semi-automatic annotation pipeline, which combines a detector, an audio classifier, human verification, and a promptable segmentation model to produce dense video-level labels for 80 categories. Taken together, the paper's claim is that high-resolution temporal modeling is what separates robust localization from brittle matching in realistic audio-visual scenes.

Load-bearing premise

The benchmark's ground-truth labels come from boxes proposed by a detector and an audio classifier and then only checked by humans, so if those machines systematically miss or mislabel a category of sound source, every model ranking built on these labels inherits that blind spot.

Editorial extensions

If this is right

  • Frame-based audio-visual localization models are systematically overrated by current image-level benchmarks, because their accuracy collapses when the active source changes over time.
  • A model with explicit temporal attention can hold its localization accuracy on source-change clips, indicating that tracking, not just spatial alignment, is a learnable and necessary component of the task.
  • The Multi-entity and Mixed-sound results suggest that temporal context lets a model distinguish the sounding instance among visually identical objects without being trained specifically on that scenario.
  • Off-screen evaluation requires per-model thresholding rather than per-frame normalization, because normalizing each frame's heatmap implicitly assumes a sound source is always visible.
  • AVATAR's 24,266 labeled frames across 80 categories give the community a video-centric evaluation standard that existing image-level datasets and clip-level segmentation benchmarks do not provide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A concrete extension would be to re-annotate a stratified sample of AVATAR frames with independent annotators and no machine-proposed boxes; if the detector-and-audio-classifier proposal step shares biases with TAVLO's encoder, part of the measured gap could be annotation, not temporal reasoning.
  • The same factorized spatial-then-temporal attention recipe could transfer directly to audio-visual segmentation and audio-visual event localization, whose current benchmarks also annotate one representative frame per clip and would benefit from dense temporal labels.
  • Since the paper explicitly leaves off-screen localization open, a natural next step is adding an explicit 'no source in frame' output trained with absence labels, rather than thresholding similarity maps that always peak somewhere.
  • The Cross-event drop, $\Delta$ between full-benchmark and Cross-event CIoU, could be adopted as a standard robustness diagnostic for any future audio-visual localization model, independent of which benchmark it is trained on.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AVATAR, a video-centric audio-visual localization (AVL) benchmark with 5,000 videos, 24,266 annotated frames, 80 categories, and four evaluation scenarios: Single-sound, Mixed-sound, Multi-entity, and Off-screen. Annotations are produced by a semi-automatic pipeline that combines YOLOv8 object detection, CAV-MAE audio classification, SAM segmentation, and human verification. The paper also proposes TAVLO, a model that encodes frame-aligned audio features and applies factorized spatial and temporal attention, trained with a modified multiple-instance contrastive loss. Experiments compare TAVLO against EZ-VSL, SLAVC, and SSL-TIE, reporting that TAVLO outperforms baselines on Mixed-sound and Multi-entity scenarios and exhibits only a small performance drop in the Cross-event subset, while baselines drop substantially. The paper concludes that explicit temporal modeling is essential for robust video-centric AVL.

Significance. If validated, the benchmark would address a genuine gap in AVL evaluation, moving beyond single-frame, single-source assumptions to dynamic, multi-source, and off-screen settings. The TAVLO architecture is a reasonable and well-motivated instantiation of video-centric modeling, and the scenario decomposition in AVATAR is a useful organizational principle. The reported Cross-event results are striking and could be influential. However, the central claims are currently under-supported: the benchmark's ground truth is partly generated by a pretrained audio-visual model without a reported reliability analysis, the Cross-event advantage is not isolated by ablations from other architectural and training differences, and the Off-screen evaluation uses a post-hoc threshold and subset change. These issues affect the validity of the benchmark as a standard and the attribution of TAVLO's performance to temporal modeling. The paper's strengths include a large-scale video annotation effort, a plausible semi-automatic pipeline, and a clear experimental structure across four scenarios.

major comments (4)
  1. [Sec. 3.1.3 and Fig. 1] The AVATAR ground-truth annotation pipeline is circular in a way that threatens the benchmark's validity. Active-sound bounding boxes are selected by CAV-MAE audio classification that filters YOLOv8 detections; human annotators then verify these proposals but do not independently propose or reject boxes from scratch. The paper reports no inter-annotator agreement, no error-rate analysis for the human verification step, and no per-category analysis of CAV-MAE failures. Since TAVLO and the baselines are all evaluated against labels produced with CAV-MAE's inductive biases, the benchmark ranking may reflect agreement with CAV-MAE rather than true audio-visual localization ability. The authors should provide an independent human annotation sample, quantify verification reliability, and analyze how often CAV-MAE filtering misses or mislabels active sources.
  2. [Sec. 5.2.1, Tables 3 and 4] The central claim that temporal modeling causes TAVLO's small Cross-event performance drop is underdetermined. TAVLO differs from EZ-VSL, SLAVC, and SSL-TIE in at least four coupled ways: multi-frame visual input, frame-aligned audio segmentation via the rectangular CNN kernel in Eq. (2), factorized spatial and temporal attention in Eqs. (7)-(11), and the mean negative response in Eq. (13) instead of EZ-VSL's max response. The paper reports no ablation removing temporal attention, no comparison with a global-audio variant, no variant with the max-negative MIL loss, and no single-frame-input variant. The audio window length experiment mentioned in Sec. 5.1 is not reported quantitatively. Without these ablations, the minimal Cross-event drop could be driven by any of the other components, such as frame-aligned audio features alone, and the attribution to 'explicit temporal modeling' is not established.
  3. [Sec. 5.2.2 and Table 2] The Off-screen evaluation is not conclusive because the metric protocol is changed after observing the initial results. In Table 2, TAVLO has the lowest TN(%) under the top-10% threshold, and the paper then reports †TN(%) computed with a top-5% threshold on only videos that contain Off-screen frames. No principled justification is given for the threshold or subset selection, and the revised numbers still leave TAVLO below several baselines (95.02 vs. 96.45, 95.84). This post-hoc adjustment makes the Off-screen comparison difficult to interpret. The authors should pre-specify the metric, report all variants, and provide confidence intervals or significance tests before claiming that models perform comparably.
  4. [Tables 2-4] No error bars, multiple seeds, or significance tests are reported for any of the quantitative results. Several scenario-level differences are small: in Single-sound, SSL-TIE(144k) reports 14.23 AUC vs. TAVLO's 14.08, and in †TN TAVLO is 95.02 vs. EZ-VSL(full)'s 95.84. Without variance estimates, it is not possible to assess whether the claimed advantages are reliable. This is especially important because the Cross-event margins, although large, are based on a single evaluation run. The authors should provide at least three seeds with standard deviations and, where appropriate, paired significance tests.
minor comments (5)
  1. [Sec. 5.1] The statement that 'direct comparison with our method may not be entirely fair' is acknowledged, but the discussion does not explain why TAVLO's smaller training set (10k vs. 144k) is a fairness concern; if anything, it disadvantages TAVLO. The real confound is the number of other differences described in the major comments.
  2. [Sec. 3.1.1] The bitrate constraint is written as '≥100 bps', which is implausibly low; presumably this is a typo for kbps. The resolution constraint '640 ×360' should specify whether this is a minimum or exact resolution.
  3. [Sec. 5.2.1] The sentence 'This conditions tests whether A VL models can effectively localize temporally varying audio sources' contains a typo ('conditions' should be 'condition').
  4. [General] There are inconsistent spellings of the object detector ('YoloV8' in Fig. 1 and 'YOLOv8' in Sec. 3.1.3), and the model names are typeset with unusual spacing (e.g., 'TA VLO', 'A V ATAR'), which should be normalized.
  5. [Sec. 5.1] The evaluation metric description says the threshold is set to the top 10% of pixel values 'across all labeled frames', but it is not clear whether this is computed per video, per frame, or globally across the dataset; please specify the exact computation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TAVLO's cross-event advantage is an empirical result with independent training data; CAV-MAE-assisted annotation is a validity concern, not a self-definitional derivation.

full rationale

The paper's central deliverables are (i) the AVATAR benchmark and (ii) TAVLO's temporal-attention model. Neither claim reduces by construction to its inputs. AVATAR labels are produced by a human-verified pipeline: YOLO boxes are filtered by CAV-MAE audio classification, then 'Human annotators verify and filter these bounding boxes by reviewing the corresponding video segment (±0.05s around the frame) to ensure that the labeled instance is indeed producing sound at that moment' (Sec. 3.1.3). That human verification is an independent grounding step, and TAVLO is trained on VGGSound without AVATAR labels, so the benchmark is not the model's training objective. TAVLO's architecture and loss are specified in Sec. 4 from factorized spatial/temporal attention and a modified MIL objective; the reported CIoU and AUC numbers are outputs of forward passes on held-out benchmark videos, not fitted parameters renamed as predictions. The CAV-MAE-guided filtering does create a possible benchmark-bias risk, because the ground-truth notion of 'active sound source' is partially shaped by a pretrained audio-visual model; however, this is a dataset-validity concern rather than a circular derivation, since CAV-MAE is neither the method under evaluation nor part of TAVLO's training signal. The strongest claim, that explicit temporal modeling causes the small Cross-event drop (Δ = -0.33 CIoU), is underdetermined by the experiments: no ablation isolates temporal attention from multi-frame visual input, frame-aligned audio segmentation, and the mean-negative MIL loss, and the paper itself notes that 'direct comparison with our method may not be entirely fair.' Underdetermination and missing ablations are attribution/completeness issues, not circularity. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz-smuggling citation is present. Therefore the paper's derivation chain is not circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the correctness of the model-assisted annotations and on the audio-visual alignment assumption in the MIL loss. There are no physics-like invented entities; the dataset and model are the claimed new artifacts. The free parameters are evaluation thresholds and sampling settings whose values are chosen by hand and, in the Off-screen case, adjusted after observing results.

free parameters (3)
  • Localization heatmap threshold = Top 10% of pixel values across all labeled frames (re-evaluated at 5% on off-screen videos for †TN)
    Hand-chosen evaluation threshold; the CIoU, AUC and TN numbers reported in Tables 2-4 are directly sensitive to this value, and it was changed after observing results in the Off-screen scenario.
  • Audio activity RMS threshold = 0.01
    Used in clip sampling (Sec. 3.1.2) to distinguish audio-active from silent intervals; changing it would alter which video segments enter the benchmark.
  • Frame sampling windows = Event window ±0.1s; exclusion ±0.7s; up to 5 frames per clip
    Hand-set parameters that determine which frames are annotated and therefore evaluated; no sensitivity analysis is provided.
assumptions (3)
  • domain assumption Each frame-level audio segment contains the sound of at least one instance in the corresponding positive visual bag.
    The contrastive MIL loss (Eqs. 12-14) requires that the positive bag contains the source; in Mixed-sound and Off-screen scenarios this assumption is weakened but still underpins the training objective.
  • domain assumption The AVATAR ground-truth annotations are accurate despite being generated by a model-assisted pipeline.
    Sec. 3.1.3 uses YOLOv8 and CAV-MAE to propose bounding boxes and then human verification; the paper provides no inter-annotator agreement or error-rate analysis.
  • standard math Self-attention with spatial and temporal factorization is a suitable fusion mechanism for audio-visual video features.
    Transformer machinery (MSA, positional encodings) is used as a background tool; the paper does not prove it is necessary or optimal, but this is a standard library component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What's Making That Sound Right Now? Video-centric Audio-Visual Localization." pith.science (2026). https://pith.science/paper/V4X6N2DN

@misc{pith2026250704667,
  author       = {Pith},
  title        = {Pith review of: What's Making That Sound Right Now? Video-centric Audio-Visual Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4X6N2DN}},
  note         = {Machine review of arXiv:2507.04667}
}
read the original abstract

Audio-Visual Localization (AVL) aims to identify sound-emitting sources within a visual scene. However, existing studies focus on image-level audio-visual associations, failing to capture temporal dynamics. Moreover, they assume simplified scenarios where sound sources are always visible and involve only a single object. To address these limitations, we propose AVATAR, a video-centric AVL benchmark that incorporates high-resolution temporal information. AVATAR introduces four distinct scenarios -- Single-sound, Mixed-sound, Multi-entity, and Off-screen -- enabling a more comprehensive evaluation of AVL models. Additionally, we present TAVLO, a novel video-centric AVL model that explicitly integrates temporal information. Experimental results show that conventional methods struggle to track temporal variations due to their reliance on global audio features and frame-level mappings. In contrast, TAVLO achieves robust and precise audio-visual alignment by leveraging high-resolution temporal modeling. Our work empirically demonstrates the importance of temporal dynamics in AVL and establishes a new standard for video-centric audio-visual localization.

Figures

Figures reproduced from arXiv: 2507.04667 by the authors.

Figure 1
Figure 1. Model-Driven Labeling and Verification Process. CAV-MAE [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Four Scenarios and Cross-event Subset Examples. (a) Single-sound: A single source (e.g., a barking dog). (b) Mixed-sound: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of Our Approach. The audio and visual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative Comparison. (a) Multi-entity: A drum performance scene where multiple drums are visible, but only the snare drum [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 29 canonical work pages

  1. [1]

    Self-supervised learning of audio-visual objects from video

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 208–224. Springer, 2020. 1

    work page 2020

  2. [2]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 6836–6846,

  3. [3]

    Soundnet: Learning sound representations from unlabeled video

    Yusuf Aytar, Carl V ondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Ad- vances in neural information processing systems , 29, 2016. 2

    work page 2016

  4. [4]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 4

    work page 2021

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 2, 3, 6

    work page 2020

  6. [6]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Na- grani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16867–16876, 2021. 1

    work page 2021

  7. [7]

    H. et al. Chen. Localizing visual sounds the hard way. In CVPR, 2021. 1, 2

    work page 2021

  8. [8]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 776–780, 2017. 3

    work page 2017

Show all 37 references
  1. [9]

    Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R

    Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representa- tions, 2023. 3

    2023

  2. [10]

    Dual mean-teacher: An unbiased semi-supervised framework for audio-visual source localization

    Yuxin Guo, Shijie Ma, Hu Su, Zhiqing Wang, Yuhao Zhao, Wei Zou, Siyang Sun, and Yun Zheng. Dual mean-teacher: An unbiased semi-supervised framework for audio-visual source localization. Advances in Neural Information Pro- cessing Systems, 36:48639–48661, 2023. 2

    2023

  3. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  4. [12]

    Deep multimodal clus- tering for unsupervised audiovisual learning

    Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clus- tering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9248–9257, 2019. 1, 2

    2019

  5. [13]

    Mix and local- ize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and local- ize: Localizing sound sources in mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022. 2

    2022

  6. [14]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910–22921, 2023. 2

    2023

  7. [15]

    Learning to visually localize sound sources from mix- tures without prior source knowledge

    Dongjin Kim, Sung Jin Um, Sangmin Lee, and Jung Uk Kim. Learning to visually localize sound sources from mix- tures without prior source knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  8. [16]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 3

    2023

  9. [17]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai...

  10. [18]

    Ex- ploiting transformation invariance and equivariance for self- supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Ex- ploiting transformation invariance and equivariance for self- supervised sound localisation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3742– 3753, 2022. 2, 6, 7

    2022

  11. [19]

    A framework for multiple-instance learning

    Oded Maron and Tomas Lozano-Perez. A framework for multiple-instance learning. In Neural Information Process- ing Systems, 1997. 2

    1997

  12. [20]

    A closer look at weakly- supervised audio-visual source localization

    Shentong Mo and Pedro Morgado. A closer look at weakly- supervised audio-visual source localization. In Advances in Neural Information Processing Systems, 2022. 1, 2, 5, 6, 7

    2022

  13. [21]

    Localizing visual sounds the easy way

    Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In European Conference on Computer Vision, pages 218–234. Springer, 2022. 2, 5, 6, 7

    2022

  14. [22]

    Audio-visual grouping net- work for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping net- work for sound localization from mixtures. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10565–10574, 2023. 2

    2023

  15. [23]

    Learn- ing representations from audio-visual spatial alignment

    Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learn- ing representations from audio-visual spatial alignment. Ad- vances in Neural Information Processing Systems, 33:4733– 4744, 2020. 1

    2020

  16. [24]

    Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5711–5720, 2024

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Can clip help sound source localization? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5711–5720, 2024. 1, 2

    2024

  17. [25]

    Audio-visual object localization and separation using low- rank and sparsity

    Jie Pu, Yannis Panagakis, Stavros Petridis, and Maja Pantic. Audio-visual object localization and separation using low- rank and sparsity. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2901–2905, 2017. 1 9

    2017

  18. [26]

    Multiple sound sources localization from coarse to fine

    Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision–ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part XX 16, pages 292–308. Springer, 2020. 2

    2020

  19. [27]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  20. [28]

    Real-time flying object detection with yolov8.arXiv preprint arXiv:2305.09972, 2023

    Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8.arXiv preprint arXiv:2305.09972, 2023. 3

    2023 arXiv

  21. [29]

    Learning to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 2

    2018

  22. [30]

    Sound source local- ization is all about cross-modal alignment

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Sound source local- ization is all about cross-modal alignment. InProceedings of the IEEE/CVF international conference on computer vision, pages 7777–7787, 2023. 1, 2, 5

    2023

  23. [31]

    Aligning sight and sound: Advanced sound source localization through audio- visual alignment

    Arda Senocak, Hyeong Sun Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Aligning sight and sound: Advanced sound source localization through audio- visual alignment. ArXiv, abs/2407.13676, 2024. 1, 2

    2024 arXiv

  24. [32]

    Learning audio-visual source localization via false negative aware contrastive learning

    Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yi- ran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware contrastive learning. InProceedings of the IEEE/CVF conference on computer vision ...

    2023

  25. [33]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on com- puter vision (ECCV), pages 247–263, 2018. 1

    2018

  26. [34]

    Scaling autoregressive video models

    Dirk Weissenborn, Oscar T ¨ackstr¨om, and Jakob Uszkoreit. Scaling autoregressive video models. In International Con- ference on Learning Representations, 2020. 4

    2020

  27. [35]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on com- puter vision (ECCV), pages 570–586, 2018. 1

    2018

  28. [36]

    Audio-visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, 2022. 1, 2

    2022

  29. [37]

    Audio- visual segmentation with semantics

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Ling- peng Kong, Meng Wang, and Yiran Zhong. Audio- visual segmentation with semantics. arXiv preprint arXiv:2301.13190, 2023. 2 10

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.