Pith. sign in

REVIEW 3 major objections 5 minor 72 references

SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding depth maps to a microphone-array camera nearly doubles accuracy for locating invisible sound sources.

desk verdict Extension of Sound3DVDet with depth cues shows strong simulated results, but an internal inconsistency in the SoundDet baseline numbers must be resolved before the headline SOTA claim can be trusted. read the letter →

arxiv 2412.16861 v2 pith:QP3WABAN submitted 2024-12-22 cs.SD cs.CVcs.MMeess.AS

Yuhang He , Sangyun Shin , Anoop Cherian , Niki Trigoni , Andrew Markham This is my paper · ORCID
classification cs.SDcs.CVcs.MMeess.AS
keywords 3Dsoundsourcelocalizationaudio-visuallearningmicrophonearrayRGB-Dmultiviewgeometrysetpredictionweaklycorrelatedeventdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SoundLoc3D aims to establish that invisible 3D sound sources, which cannot be seen but lie on an object's surface, can be reliably localized and classified by combining a four-microphone array with multiview RGB-D images. The paper frames the task as set prediction: each view's audio features generate initial source queries, and multiview RGB-D cues refine those queries to lie on physical surfaces. On a simulated benchmark built from indoor scenes with rendered acoustics, the method reports mAP of 0.518 versus 0.309 for the closest audio-visual prior, with mean localization error dropping from 0.586 m to 0.320 m. A sympathetic reader would care because this addresses a real gap: many fault sounds, like gas leaks or failing fans, are invisible but have a physical location that depth geometry can pin down.

What carries the argument

The load-bearing mechanism is the depth-informed closeness loss (Eq. 12). For each query predicted from one view, SoundLoc3D projects the predicted 3D point into each other view's depth map, reads the depth value along that projection ray, back-projects to get a depth-informed 3D position, and takes the centroid across views. The loss penalizes the L2 distance between the prediction and that centroid once it exceeds 0.3 m, thereby pulling the prediction toward the object surface. This is supported by the cross-view consistency loss (Eq. 13), which ties predictions of the same source across views, and by a pretrained detector-free local feature matcher that supplies multiview appearance features at projected source locations, making visual consistency usable even on texture-homogeneous surfaces.

What would settle it

Run the paper's wall-scene dataset with a systematic +0.4 m bias applied to every depth map: if mean localization error rises by roughly 0.4 m, the depth loss is directly pulling predictions onto the biased surface, while if it stays near 0.32 m the loss is more robust than the on-surface assumption would predict.

Watch

Extended reading notes

Core claim

The central claim is that depth maps are the key missing modality for weakly correlated audio-visual sound-source localization. SoundLoc3D treats the problem as set prediction: a query generator produces initial sound-source queries from each view's log-mel spectrograms and GCC-PHAT features; a feature mixer then aggregates multiview RGB appearance features at the projected position of each query; and a decoder produces the final 3D position and class. Two losses carry the load: a depth-informed closeness loss that penalizes the distance between the predicted source and the centroid of its multiview depth back-projections beyond 0.3 m, and a cross-view consistency loss that keeps the same source's predictions from different views in agreement. The paper reports that this pipeline outperforms seven prior methods, with mAP 0.518, mAR 0.999, and mALE 0.320, and is robust to added camera-pose noise and white ambient noise. The conclusion is that depth's direct surface geometry, not just RGB appearance, is what makes invisible sources localizable.

Load-bearing premise

The depth-informed closeness loss assumes the true sound source lies on the depth-reconstructed object surface; if the source is off-surface, occluded, or the depth is biased, this cue can pull predictions onto the wrong surface.

Editorial extensions

If this is right

  • If the reported numbers hold, an acoustic-camera rig can pinpoint invisible sources to an average error of about 0.32 m in simulated indoor scenes, below the 0.5 m true-positive threshold.
  • Depth contributes more than RGB appearance: removing the depth loss drops mAP from 0.518 to 0.472 and raises mALE to 0.457, while removing RGB keeps mAP at 0.498, so geometry is the primary cross-modal cue.
  • The method scales from one to six views, with mAP rising from 0.412 to 0.522 and then saturating, meaning only a few viewpoints are needed for most of the benefit.
  • The framework keeps most of its performance under practical distortions: mAP stays at 0.480 under 0.3 rad camera-pose noise and at 0.467 under 5 dB white noise, while comparators drop further.
  • Because it treats detection as set prediction, the method handles a variable number of sources without non-maximum suppression and remains stable when the number of sound classes grows from 5 to 9.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the on-surface assumption carries over to real rooms only approximately, a natural extension is a learned or view-dependent tolerance in the depth closeness loss; the paper's fixed 0.3 m threshold is a stand-in for that uncertainty.
  • The robustness study covers white noise and pose jitter but not biased or missing depth; a test with a systematic depth shift would separate the depth cue's value from its sensitivity to depth error.
  • The same query-refinement loop could be inverted: audio could disambiguate which of several depth-surfaced objects emitted a sound in occluded scenes, a case the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SoundLoc3D proposes a multimodal acoustic-camera pipeline for joint 3D localization and classification of invisible sound sources. The method treats the task as set prediction: initial queries are generated from single-view microphone-array features (log-mel and GCC-Phat), then refined by aggregating multiview RGB features from a pretrained LoFTR model, a depth-informed closeness loss that pulls predictions toward depth-reconstructed object surfaces, and a cross-view consistency loss. Experiments on a large simulated multiview RGB-D and microphone-array dataset, created following the Sound3DVDet pipeline, report that SoundLoc3D outperforms seven prior methods (mAP 0.518, mAR 0.999, mALE 0.320 versus Sound3DVDet at 0.309, 0.998, 0.586). The paper also includes ablations on view number, microphone number, and loss components, plus robustness tests with acoustic noise and camera-pose noise.

Significance. If the reported results are correct, SoundLoc3D makes a useful empirical contribution to audio-visual 3D sound-source localization in the weakly correlated setting. The method is lightweight (3.8M parameters), simple to describe, and the paper includes several informative ablations plus an explicit release of code and data in the supplementary material. The depth-informed closeness loss and cross-view consistency loss are natural ideas, and the comparison against audio-only baselines and the prior Sound3DVDet system is thorough. However, the evaluation is entirely synthetic, and one per-object baseline table is internally inconsistent with the headline numbers; until that inconsistency is resolved, the claimed state-of-the-art margin over SoundDet cannot be trusted. The robustness claims also cover only certain noise types, not the depth-degradation scenarios that the method's surface assumption would most fear.

major comments (3)
  1. [Appendix E.3, Table 14; Table 2] The SoundDet row in Table 14 is irreconcilable with the same method's row in Table 2 and per-class Table 13. Table 14 reports SoundDet mAP/mAR/mALE of 0.677/0.815/0.122 on Table, 0.682/0.813/0.110 on Ceiling, and 0.660/0.833/0.108 on Door, whereas Table 2 and Table 13 report overall mAP 0.120, mAR 0.674, mALE 0.823 and per-class AP values around 0.119. The stated standard deviations (at most 0.02) cannot account for a change of about 0.55 in mAP. Since the paper's central claim is a large improvement over SoundDet, this inconsistency must be resolved: either the per-object table is misaligned or erroneous, or the headline comparison is wrong. Please correct and re-verify all tables before resubmission.
  2. [Section 4.3, Eq. (12)] The robustness claim 'robust to RGB-D measurement inaccuracy' is supported only by adding Gaussian noise to camera rotation parameters (Table 6), not by perturbing depth values. The depth-informed closeness loss in Eq. (12) directly assumes the sound source lies on the depth-reconstructed object surface; depth bias, missing-depth regions, or off-surface sources could pull predictions to the wrong surface. Please add experiments with depth noise/bias and missing depth, or restrict the claim to camera-pose inaccuracy.
  3. [Section 4, Dataset Creation; Appendix C] The appendix states that the dataset used here is not exactly the same as the original Sound3DVDet dataset because views without depth maps were filtered out. Since all methods are retrained and evaluated on the same filtered set, the comparison is internally fair, but the main text should state this explicitly and should also report the effect of the filtering on Sound3DVDet's performance relative to its original paper. Without this, readers cannot judge how much of the reported gain over Sound3DVDet comes from the depth-modality change versus the dataset re-filtering.
minor comments (5)
  1. [Eq. (10)] The second decoding step writes '(Pupdate,i, Cupdate,i) = D(Qinit,i)' but it should almost certainly be D(Qupdate,i); otherwise the 'updated' queries are never decoded.
  2. [Eq. (13)] The summation is written as 'sum_{j=i}' with the condition i ≠ j; the index should be j ≠ i (or j > i) to avoid a self-pair and to match the stated combination count C = binom(N,2).
  3. [Appendix E, Table 14 caption] The appendix says all standard deviations are within 0.02, while the main text says all are ≤0.010; please make these statements consistent.
  4. [Table 7] The columns in Table 7 are ambiguous: the header 'Num Sound3DVDet Ours' with rows like '5 0.267 0.497' makes it unclear which numbers are mAP values for which method and which are class counts; please reformat.
  5. [Section 3.6] The definition of Pcentroid,i should state how depth back-projection handles invalid or zero-depth pixels, since the depth-informed loss could otherwise be computed over incomplete data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed gains are empirical and benchmarked against external and prior baselines; the depth-informed loss encodes the paper's stated surface assumption rather than deriving the conclusion from itself.

full rationale

The paper makes no first-principles derivation whose conclusion is already contained in its premises. It is an empirical learning method: initial audio queries are decoded, refined with multiview RGB LoFTR features, and trained with cross-view and depth-informed losses, then evaluated against seven baselines on a held-out simulated test set. The depth-informed closeness loss (Eq. 12) is a supervised inductive bias: it assumes the benchmark's defining condition that sources lie on object surfaces, and it pulls predictions toward the multiview depth centroid during training. That is a modeling assumption aligned with the dataset, not a tautological reduction; the ablations (SL3D noDepth, SL3D noRGBD) show performance changes that could in principle have gone the other way. The comparison includes self-citations ([26], [27], [28]) as baselines, but these are not load-bearing justifications of the central claim; the claim rests on measured metrics, and external methods (SELDNet, EIN-v2, SALSA, SALSA-Lite) plus ablations provide independent content. A separate correctness concern, outside circularity, is that Table 14 lists SoundDet with mAP/mAR/mALE values on Table/Ceiling/Door (e.g., 0.677/0.815/0.122) that are irreconcilable with the same method's overall row 0.120/0.674/0.823 in Table 2 and per-class rows in Table 13; this undermines the reported margin over SoundDet but is a reporting inconsistency, not a circular step, and no equation in the paper reduces a prediction to a fitted input.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a synthetic benchmark, the geometric assumption that sources lie on depth-reconstructed surfaces, known camera poses, and the fidelity of the acoustic simulator. No new physical entities are introduced; the acoustic-camera rig is a hardware configuration, not a postulated entity.

free parameters (4)
  • Depth loss margin sigma = 0.3 m
    Threshold in Eq. 12 beyond which the depth-informed closeness loss is incurred. Set by hand to tolerate depth inaccuracies; directly controls how strongly predictions are pulled toward depth surfaces.
  • Number of queries q = 16
    Number of DETR-style sound source queries in the set prediction head, fixed across all scenes.
  • Loss weights lambda_1, lambda_2, lambda_3 = 1.0 each
    Weights in Eq. 14 for the bipartite, depth, and cross-view losses. No tuning or sensitivity analysis is reported.
  • Inter-source separation in dataset = 0.3 m minimum
    Data generation places sound sources at least 0.3 m apart, which determines task difficulty and interacts with the evaluation distance thresholds.
assumptions (6)
  • domain assumption Sound sources lie exactly on physical object surfaces.
    Problem definition in Sec. 3.1 states each source is on an object surface; this justifies the depth-informed closeness loss and multiview visual consistency constraint.
  • domain assumption Camera poses are known and the rig is pre-calibrated and synchronized.
    Cross-view rigid transforms (Eq. 5) and perspective projections (Eq. 6) require accurate extrinsic and intrinsic parameters for every view.
  • domain assumption A coarse spatial prior about the sound source location is available.
    Sec. 3.1 assumes the method knows roughly where sources are (e.g., gas pipes along walls) and places the camera about 3 m away with no visual occlusion.
  • domain assumption SoundSpaces 2.0 and Matterport3D simulations faithfully model real acoustic propagation and visual appearance.
    All training and evaluation use synthetic data; realism of reverberation, source directivity, and depth sensor noise is assumed.
  • domain assumption LoFTR pretrained features provide valid visual correspondences on both texture-homogeneous and texture-discriminative surfaces.
    The RGB-informed cue in Sec. 3.4 relies on LoFTR matching of projected source points across views, which is stated in Appendix A.
  • standard math Projections of an on-surface 3D point are visually similar across views under the Lambertian assumption.
    This classical multiview geometry property is used in Sec. 3.4 and Fig. 3A to justify the visual consistency constraint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera." pith.science (2026). https://pith.science/paper/QP3WABAN

@misc{pith2026241216861,
  author       = {Pith},
  title        = {Pith review of: SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QP3WABAN}},
  note         = {Machine review of arXiv:2412.16861}
}
read the original abstract

Accurately localizing 3D sound sources and estimating their semantic labels -- where the sources may not be visible, but are assumed to lie on the physical surface of objects in the scene -- have many real applications, including detecting gas leak and machinery malfunction. The audio-visual weak-correlation in such setting poses new challenges in deriving innovative methods to answer if or how we can use cross-modal information to solve the task. Towards this end, we propose to use an acoustic-camera rig consisting of a pinhole RGB-D camera and a coplanar four-channel microphone array~(Mic-Array). By using this rig to record audio-visual signals from multiviews, we can use the cross-modal cues to estimate the sound sources 3D locations. Specifically, our framework SoundLoc3D treats the task as a set prediction problem, each element in the set corresponds to a potential sound source. Given the audio-visual weak-correlation, the set representation is initially learned from a single view microphone array signal, and then refined by actively incorporating physical surface cues revealed from multiview RGB-D images. We demonstrate the efficiency and superiority of SoundLoc3D on large-scale simulated dataset, and further show its robustness to RGB-D measurement inaccuracy and ambient noise interference.

Figures

Figures reproduced from arXiv: 2412.16861 by the authors.

Figure 1
Figure 1. SoundLoc3D problem setup: Visually invisible sound sources freely lie on physical object’s surface and are emitting sound, A: We use an acoustic-camera to record Mic-Array signal and RGB-D images from multiview. SoundLoc3D incorporates multiview crossmodal RGB images, depth maps and Mic-Array signal to jointly localize source position p and semantic label c. and thus rely solely on acoustic signals, and 2) methods t… view at source ↗
Figure 2
Figure 2. SoundLoc3D Pipeline. The RGB image is first pre-processed by a feature matching aware pre-trained model to get an embedding (LoFTR), Mic-Array signal feature is extracted by stacking Log-Mel scale TF and GCC-Phat features. The query generator G is applied to get the initial queries, which are further fed to query decoder D to aggregate crossview RGB image informed sound source cues. The queries after aggregation is … view at source ↗
Figure 3
Figure 3. Sound Source Cue from Multiview RGB-D images and Crossview Consistency: A. While only “on the surface” sound source’s projections onto multiview RGB images are guaranteed to be visually similar, either above or below the surface sound sources are much less likely to be visually similar. B. The closer of predicted sound source to the object surface, the smaller of its distance to multiview depth maps informed source … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Localization Result Visualization: We visualize the sound source localization result in the 3D visual space by different methods as well as its ground truth position. Zoom in for better visualization. We provide data and visualization code and in Supplementary material…
Figure 6
Figure 6. Figure 6: Matching points visualization. A. Two RGB images from different views. They contain large texture homogeneous area. B. LoFTR manages to predict dense matching points even on these texture homogeneous areas. We adopt LoFTR [52] to extract RGB image feature, which provid…
Figure 7
Figure 7. Figure 7: SoundLoc3D experiment data visualization: We visualize the sample data we used in our experiment. C. More Discussion on Dataset Creation In the supplementary material, we provide the statistics of the created large dataset in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: More qualitative result: We visualize the localization result for one sound source in different visual scenes. We also provide the visualization source code and data for more directive visualization [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 68 canonical work pages

  1. [1]

    Sound Event Detection Using Spatial Features and Convolutional Recurrent Neural Network

    Sharath Adavanne, Pasi Pertil¨a, and Tuomas Virtanen. Sound Event Detection Using Spatial Features and Convolutional Recurrent Neural Network. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017. 1, 2, 4, 6, 7, 3, 5

    work page 2017

  2. [2]

    The Conversation: Deep Audio-Visual Speech Enhance- ment

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisser- man. The Conversation: Deep Audio-Visual Speech Enhance- ment. arXiv preprint arXiv:1804.04121, 2018. 2

    arXiv 2018

  3. [3]

    C-3PO: Towards Rotation Equivariant Feature Detection and Description

    Piyush Bagad, Floor Eijkelboom, Mark Fokkema, Danilo de Goede, Paul Hilders, and Miltiadis Kofinas. C-3PO: Towards Rotation Equivariant Feature Detection and Description. In European Conference on Computer Vision Workshops (EC- CVW), 2022. 2

    work page 2022

  4. [4]

    SIFT Matching by Context Exposed

    Fabio Bellavia. SIFT Matching by Context Exposed. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2022. 2

    work page 2022

  5. [5]

    M. S. Brandstein and H. F. Silverman. A Robust Method for Speech Signal Time-Delay Estimation in Reverberant Rooms. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1997. 4

    work page 1997

  6. [6]

    Busso, S

    C. Busso, S. Hernanz, Chi-Wei Chu, Soon il Kwon, Sung Lee, P.G. Georgiou, I. Cohen, and S. Narayanan. Smart Room: Participant and Speaker Localization and Identification. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2005. 1

    work page 2005

  7. [7]

    An Improved Event- Independent Network for Polyphonic Sound Event Localiza- tion and Detection

    Yin Cao, Turab Iqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D Plumbley. An Improved Event- Independent Network for Polyphonic Sound Event Localiza- tion and Detection. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021. 1, 2, 4, 6, 7, 8, 3, 5

    work page 2021

  8. [8]

    Event-Independent Network for Polyphonic Sound Event Localization and Detection

    Yin Cao, Turab Iqbal, Qiuqiang Kong, Yue Zhong, Wenwu Wang, and Mark D Plumbley. Event-Independent Network for Polyphonic Sound Event Localization and Detection. In DCASE Workshop, 2020. 2, 4

    work page 2020

Show all 72 references
  1. [9]

    End-to- end object detection with Transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with Transformers. In European Confer- ence on Computer Vision (ECCV), 2020. 2, 3

    2020

  2. [10]

    Matterport3D: Learning from RGB- D Data in Indoor Environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB- D Data in Indoor Environments. International Conference on 3D Vision (3DV), 2017. 6

    2017

  3. [11]

    Learning audio-visual dereverberation

    Changan Chen and Wei Sun and David Harwath and Kris- ten Grauman. Learning audio-visual dereverberation. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023. 2

    2023

  4. [12]

    SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

    Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robin- son, and Kristen Grauman. SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning. In Conference on Neural Information Processing Systems Track ...

    2022

  5. [13]

    Polar Parametrization for Vision-Based Surround-View 3D Detection

    Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu. Polar Parametrization for Vision-Based Surround-View 3D Detection. arXiv preprint arXiv:2206.10965, 2022. 2

    2022 arXiv

  6. [14]

    Guide Local Feature Matching by Overlap Estimation

    Ying Chen, Dihe Huang, Shang Xu, Jianlin Liu, and Yong Liu. Guide Local Feature Matching by Overlap Estimation. In Proceedings of the AAAI Conference on Artificial Intelli- gence (AAAI), 2022. 2

    2022

  7. [15]

    Graph-DETR3D: rethinking overlapping regions for multi-view 3D object detection

    Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qin- hong Jiang, and Feng Zhao. Graph-DETR3D: rethinking overlapping regions for multi-view 3D object detection. In Proceedings of the 30th ACM International Conference on Multimedia, 2022. 4

    2022

  8. [16]

    Chronister, Tessa A

    Lauren M. Chronister, Tessa A. Rhinehart, Aidan Place, and Justin Kitzes. An annotated set of audio recordings of eastern north american birds containing frequency, time, and species information, 2021. 1

    2021

  9. [17]

    Empirical evaluation of gated recurrent neu- ral ntworks on sequence modelling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neu- ral ntworks on sequence modelling. In Advances Neural Information Processing System (NeurIPS), 2014. 7

    2014

  10. [18]

    Imagenet: A Large-Scale Hierarchical Image Database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A Large-Scale Hierarchical Image Database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 3

    2009

  11. [19]

    Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-visual Model for Speech Sep- aration

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-visual Model for Speech Sep- aration. arXiv preprint arXiv:1804.03619, 2018. 2

    2018 arXiv

  12. [20]

    Seeing through Noise: Visually driven Speaker Separation and Enhancement

    Aviv Gabbay, Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. Seeing through Noise: Visually driven Speaker Separation and Enhancement. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), 2018. 2

    2018

  13. [21]

    Learning to Separate Object Sounds by Watching Unlabeled Video

    Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to Separate Object Sounds by Watching Unlabeled Video. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 2

    2018

  14. [22]

    A study of the complexity and accuracy of direction of arrival estimation methods based on gcc-phat for a pair of close microphones

    Francois Grondin, James Glass, Iwona Sobieraj, and Plumb- ley Mark D. A study of the complexity and accuracy of direction of arrival estimation methods based on gcc-phat for a pair of close microphones. In Proceedings of the Detection and Classification of Acoustic Scenes and ...

  15. [23]

    L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environment

    Eric Guizzo, Christian Marinoni, Marco Pennese, Xinlei Ren, Xiguang Zheng, Chen Zhang, Bruno Masiero, Aurelio Uncini, and Danilo Comminiello. L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environment. In IEEE International Conference on Acoustics, Speech and Si...

    2022

  16. [24]

    Identity mappings in deep residual networks

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016. 1, 3

    2016

  17. [25]

    Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space

    Yuhang He, Irving Fang, Yiming Li, Rushi Bhavesh Shah, and Chen Feng. Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space. In Robotics: Science and Systems (RSS), 2023. 6

    2023

  18. [26]

    SoundDoA: Learn Sound Source Direction of Arrival and Semantics from Sound Raw Waveforms

    Yuhang He and Andrew Markham. SoundDoA: Learn Sound Source Direction of Arrival and Semantics from Sound Raw Waveforms. In Interspeech, 2022. 1, 2, 6, 7, 3, 5

    2022

  19. [27]

    Sound3DVDet: 3D Sound Source Detec- tion Using Multiview Microphone Array and RGB Images

    Yuhang He, Sangyun Shin, Anoop Cherian, Niki Trigoni, and Andrew Markham. Sound3DVDet: 3D Sound Source Detec- tion Using Multiview Microphone Array and RGB Images. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 5496–5507, January

  20. [28]

    SoundDet: Polyphonic Moving Sound Event Detection and Localiza- tion from Raw Waveform

    Yuhang He, Niki Trigoni, and Andrew Markham. SoundDet: Polyphonic Moving Sound Event Detection and Localiza- tion from Raw Waveform. In International Conference on Machine Learning (ICML), 2021. 1, 2, 6, 7, 3, 5

    2021

  21. [29]

    Psychophysiological effect of immersive spatial audio experience enhanced using sound field synthesis

    Yasuhide Hyodo, Chihiro Sugai, Junya Suzuki, Masafumi Takahashi, Masahiko Koizumi, Asako Tomura, Yuki Mit- sufuji, and Yota Komoriya. Psychophysiological effect of immersive spatial audio experience enhanced using sound field synthesis. In International Conference on Affective...

    2021

  22. [30]

    Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2:83–97, 1955. 5, 6

    1955

  23. [31]

    Deeply-Supervised Nets

    Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-Supervised Nets. In In- ternational Conference on Artificial Intelligence and Statis- tics (AISTATS), 2015. 6

    2015

  24. [32]

    Dual- resolution Correspondence Networks

    Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Dual- resolution Correspondence Networks. Advances in Neural Information Processing Systems (NeurIPS), 2020. 2

    2020

  25. [33]

    SIFT Flow: Dense Correspondence across Scenes and its Applications

    Ce Liu, Jenny Yuen, and Antonio Torralba. SIFT Flow: Dense Correspondence across Scenes and its Applications. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2010. 2

    2010

  26. [34]

    Extremely Dense Point Correspondences Using a Learned Feature Descriptor

    Xingtong Liu, Yiping Zheng, Benjamin Killeen, Masaru Ishii, Gregory D Hager, Russell H Taylor, and Mathias Unberath. Extremely Dense Point Correspondences Using a Learned Feature Descriptor. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  27. [35]

    PETR: Position Embedding Transformation for Multi-View 3D Object Detection

    Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETR: Position Embedding Transformation for Multi-View 3D Object Detection. European Conference on Computer Vision (ECCV), 2022. 2, 4

    2022

  28. [36]

    Petrv2: A Unified Framework for 3D Perception from Multi-Camera Images

    Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A Unified Framework for 3D Perception from Multi-Camera Images. arXiv preprint arXiv:2206.01256, 2022. 2, 4

    2022 arXiv

  29. [37]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representation (ICLR), 2019. 7

    2019

  30. [38]

    Listen and Look: Audio-visual Matching assisted Speech Source Separation

    Rui Lu, Zhiyao Duan, and Changshui Zhang. Listen and Look: Audio-visual Matching assisted Speech Source Separation. In IEEE Signal Processing Letters, 2018. 2

    2018

  31. [39]

    A unified audio-visual learning framework for localization, separation, and recog- nition

    Shentong Mo and Pedro Morgado. A unified audio-visual learning framework for localization, separation, and recog- nition. In International Conference on Machine Learn- ing (ICML), 2023. 1

    2023

  32. [40]

    Audio-visual grouping net- work for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping net- work for sound localization from mixtures. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR),

  33. [41]

    Face Landmark-based Speaker-independent Audio-visual Speech Enhancement in Multi-talker Environments

    Giovanni Morrone, Sonia Bergamaschi, Luca Pasa, Luciano Fadiga, Vadim Tikhanoff, and Leonardo Badino. Face Landmark-based Speaker-independent Audio-visual Speech Enhancement in Multi-talker Environments. In IEEE Interna- tional Conference on Acoustics, Speech and Signal Proces...

    2019

  34. [42]

    Watcharasupat, Ngoc Khanh Nguyen, Douglas L

    Thi Ngoc Tho Nguyen, Karn N. Watcharasupat, Ngoc Khanh Nguyen, Douglas L. Jones, and Woon-Seng Gan. SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Poly- phonic Sound Event Localization and Detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing ,

  35. [43]

    Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Atten- tion

    Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Atten- tion. In Proceedings of the IEEE/CVF Conference on Winter Conference on Application of Computer Vision (WACV), 2022. 2

    2022

  36. [44]

    PyTorch: An Imper- ative Style, High-Performance Deep Learning Library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

  37. [45]

    Audio-visual Object Localization and Separation using Low- rank and Sparsity

    Jie Pu, Yannis Panagakis, Stavros Petridis, and Maja Pantic. Audio-visual Object Localization and Separation using Low- rank and Sparsity. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017. 2

    2017

  38. [46]

    Effi- cient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions

    Ignacio Rocco, Relja Arandjelovi ´c, and Josef Sivic. Effi- cient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions. In European Conference on Computer Vision (ECCV), 2020. 2

    2020

  39. [47]

    Superglue: Learning Feature Match- ing with Graph Neural Networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning Feature Match- ing with Graph Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  40. [48]

    Structure-from-motion revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2

    2016

  41. [49]

    Learning to Localize Sound Source in Visual Scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to Localize Sound Source in Visual Scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 1, 2

    2018

  42. [50]

    Robust Audio Surveil- lance Using Spectrogram Image Texture Feature

    Roneel V Sharan and Tom J Moir. Robust Audio Surveil- lance Using Spectrogram Image Texture Feature. In Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2015. 1

    2015

  43. [51]

    A Detector- oblivious Multi-arm Network for Keypoint Matching

    Xuelun Shen, Qian Hu, Xin Li, and Cheng Wang. A Detector- oblivious Multi-arm Network for Keypoint Matching. IEEE Transactions on Image Processing, 2023. 2

    2023

  44. [52]

    LoFTR: Detector-Free Local Feature Match- ing with Transformers

    Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-Free Local Feature Match- ing with Transformers. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2, 4, 6, 1, 3

    2021

  45. [53]

    Jones, Karn N

    Thi Ngoc Tho Nguyen, Douglas L. Jones, Karn N. Watcha- rasupat, Huy Phan, and Woon-Seng Gan. SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Lo- calization and Detection with Microphone Arrays. In IEEE International Conference on Acoustics, Speech and Signa...

    2022

  46. [54]

    Audio-visual Event Localization in Unconstrained Videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual Event Localization in Unconstrained Videos. In Proceedings of the European Conference on Com- puter Vision (ECCV), 2018. 2

    2018

  47. [55]

    GLU- Net: Global-local Universal Network for Dense Flow and Correspondences

    Prune Truong, Martin Danelljan, and Radu Timofte. GLU- Net: Global-local Universal Network for Dense Flow and Correspondences. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  48. [56]

    DISK: Learning Local Features with Policy Gradient

    Michał Tyszkiewicz, Pascal Fua, and Eduard Trulls. DISK: Learning Local Features with Policy Gradient. Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 4

    2020

  49. [57]

    Time-domain Generalized Cross Correlation Phase Transform Sound Source Localization for Small Microphone Arrays

    Bert Van Den Broeck, Alexander Bertrand, Peter Karsmak- ers, Bart Vanrumste, Hugo Van hamme, and Marc Moonen. Time-domain Generalized Cross Correlation Phase Transform Sound Source Localization for Small Microphone Arrays. In The 5th European DSP in Education and Research Conf...

    2012

  50. [58]

    Gomez, and Lukasz Kaiser

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Uszkor- eit, Llion Jones Jone, Aidan N. Gomez, and Lukasz Kaiser. Attention is All You Need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017. 5, 7

    2017

  51. [59]

    A 3-d immersive synthe- sizer for environmental sounds

    Charles Verron, Mitsuko Aramaki, Richard Kronland- Martinet, and Gr ´egory Pallone. A 3-d immersive synthe- sizer for environmental sounds. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 2010. 1

    2010

  52. [60]

    Wang and P

    H. Wang and P. Chu. V oice Source Localization for Automatic Camera Pointing System in Videoconferencing. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1997. 1

    1997

  53. [61]

    The ustc-iflytek system for sound event localization and de- tection of dcase2020 challenge

    Qing Wang, Huaxin Wu, Zijun Jing, Feng Ma, Yi Fang, Yux- uan Wang, Tairan Chen, Jia Pan, Jun Du, and Chin-hui Lee. The ustc-iflytek system for sound event localization and de- tection of dcase2020 challenge. In DCASE workshop, 2020. 4

    2020

  54. [62]

    Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, , and Justin M. Solomon. DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries. In The Conference on Robot Learning, 2021. 2, 3, 4

    2021

  55. [63]

    Wessels, Jeroen v

    Peter W. Wessels, Jeroen v. Sande, and Frits Van der Eerden. Detection and localization of impulsive sound events for envi- ronmental noise assessment. The Journal of the Acoustical Society of America, 2017. 1

    2017

  56. [64]

    DeepMatcher: A Deep Transformer-based Network for Ro- bust and Accurate Local Feature Matching

    Tao Xie, Kun Dai, Ke Wang, Ruifeng Li, and Lijun Zhao. DeepMatcher: A Deep Transformer-based Network for Ro- bust and Accurate Local Feature Matching. arXiv preprint arXiv:2301.02993, 2023. 4

    2023 arXiv

  57. [65]

    A Proposal-based Paradigm for Self-supervised Sound Source Localization in Videos

    Hanyu Xuan, Zhiliang Wu, Jian Yang, Yan Yan, and Xavier Alameda-Pineda. A Proposal-based Paradigm for Self-supervised Sound Source Localization in Videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1

    2022

  58. [66]

    Learning Soft Estimator of Keypoint Scale and Ori- entation with Probabilistic Covariant Loss

    Pei Yan, Yihua Tan, Shengzhou Xiong, Yuan Tai, and Yan- sheng Li. Learning Soft Estimator of Keypoint Scale and Ori- entation with Probabilistic Covariant Loss. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR),

  59. [67]

    Permutation Invariant Training of Deep Models for Speaker-Independent Multi-Talker Speech Separation

    Dong Yu, Morten Kolbæ k, Zheng-Hua Tan, and Jesper Jensen. Permutation Invariant Training of Deep Models for Speaker-Independent Multi-Talker Speech Separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017. 7

    2017

  60. [68]

    High-frequency stereo match- ing network

    Haoliang Zhao, Huizhou Zhou, Yongjun Zhang, Jie Chen, Yitong Yang, and Yong Zhao. High-frequency stereo match- ing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1327–1336, June 2023. 2

    2023

  61. [69]

    Visually guided sound source sep- aration and localization using self-supervised motion repre- sentations

    Lingyu Zhu and Esa Rahtu. Visually guided sound source sep- aration and localization using self-supervised motion repre- sentations. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022. 1

    2022

  62. [70]

    on-the- surface

    Shengjie Zhu and Xiaoming Liu. Pmatch: Paired masked image modeling for dense geometric matching. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2023. 4 Appendix A. More Discussion on LoFTR A.TwoImagesfromDifferentViewpoints B.MatchingPointsfoundby...

    2023

  63. [71]

    We test the performance of replacing LoFTR with widely used ImageNet [18] pre-trained ResNet50 [24] as image feature extractor

    LoFTR vs ResNet LoFTR [52] is better suited to our problem setup as it uses the projections of sound source locations with visual consistency. We test the performance of replacing LoFTR with widely used ImageNet [18] pre-trained ResNet50 [24] as image feature extractor. This v...

  64. [72]

    In Sound3DLoc, we jointly train both the initial queries and updated queries

    Without Deep Supervision. In Sound3DLoc, we jointly train both the initial queries and updated queries. We ablate the performance without deep supervision. To this end, we remove the loss (Eqn. (14) in the main paper) added to the initial queries (SL3D noDeepSup). From Table 1...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.