{"id":"2c55df3f-3032-4209-bd00-bc4544874cf3","arxiv_id":"2412.16861","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding depth maps from an RGB-D camera to multiview microphone-array signals markedly improves 3D localization and classification of visually invisible sound sources in simulated indoor scenes.","lead":"SoundLoc3D is a neural network that finds the 3D position and type of invisible sound sources, such as gas leaks or ringing phones, using a camera and a four-microphone array from several viewpoints. It reports large gains over prior methods on a large simulated indoor dataset, but has not yet been demonstrated with real hardware.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SoundDet baseline values in Table 14 contradict Table 2 by ~0.56 mAP; the claimed SOTA margin over SoundDet cannot be trusted until this inconsistency is resolved.","rationale":"Good-faith summary: SoundLoc3D is a coherent extension of Sound3DVDet, adding a depth-informed closeness loss and cross-view consistency; the ablations suggest each component contributes, and the 3.8M parameter model is lightweight. The simulated benchmark is a reasonable first evaluation. However, the paper's central claim is an empirical margin, and that margin is only as trustworthy as the baseline table. The inconsistency in Table 14 is not a theoretical disagreement or a matter of consensus; it is an internal numerical contradiction within the manuscript. It directly changes whether the claimed improvement over SoundDet exists. This is more immediately load-bearing than the also valid depth-robustness concern raised by the reader: even if depth cues transfer perfectly, an incorrect baseline number would invalidate the comparison. I therefore recommend no change to the reader's CONDITIONAL verdict, but add this as a required condition: the authors must correct or reproduce Table 14 and resolve the discrepancies across Tables 2/13/14, and ideally release the data-creation code and per-view predictions. I do not read the inconsistency as evidence of bad faith; a row or column alignment error is the most plausible explanation, but it still needs verification. Agreement with the reader is partial: the reader named the depth-surface assumption as the weakest external-validity link, which remains real, but the internal table contradiction is the sharpest correctness risk for the headline claim.","tokens_in":22265,"tokens_out":10443,"duration_ms":90285,"concrete_test":"Obtain the released code and test-set predictions (or regenerate the benchmark with the released data-creation code once available) and rerun SoundDet using the paper's official evaluation script, reporting mAP/mAR/mALE per object category and overall. Compare exactly with Table 2, Table 13, and Table 14. If SoundDet's per-object mAP on Table/Ceiling/Door is around 0.12 with mAR around 0.67 and mALE around 0.82, Table 14 is erroneous and the Table 2 margin stands. If SoundDet's per-object mAP is around 0.66-0.68, the headline comparison in Table 2 is wrong and SoundLoc3D is not superior to SoundDet on those categories. This single reproduction check also verifies column ordering and evaluation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SoundLoc3D outperforms all seven prior methods, in particular SoundDet, by a large margin (Table 2: SoundLoc3D mAP 0.518 vs SoundDet 0.120, mAR 0.999 vs 0.674, mALE 0.320 vs 0.823). However, the appendix per-object table (Table 14) reports SoundDet with mAP/mAR/mALE of 0.677/0.815/0.122 on Table objects, 0.682/0.813/0.110 on Ceiling, and 0.660/0.833/0.108 on Door. These numbers are irreconcilable with the same method's overall row in Table 2 and per-class row in Table 13 (AP around 0.119, AR around 0.67, ALE around 0.82). The reported standard deviations (at most 0.010) cannot explain a 0.55 change in mAP. If the Table 14 numbers were correct, SoundDet would outperform SoundLoc3D on those object subsets, directly contradicting the headline SOTA claim; if Table 14 is a typographical or column-alignment error, the appendix evidence still needs correction. Since the paper's main result is an empirical comparison, this internal inconsistency is the most load-bearing unresolved issue: it determines whether the claimed improvement over a key baseline exists at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SoundLoc3D proposes a multimodal acoustic-camera pipeline for joint 3D localization and classification of invisible sound sources. The method treats the task as set prediction: initial queries are generated from single-view microphone-array features (log-mel and GCC-Phat), then refined by aggregating multiview RGB features from a pretrained LoFTR model, a depth-informed closeness loss that pulls predictions toward depth-reconstructed object surfaces, and a cross-view consistency loss. Experiments on a large simulated multiview RGB-D and microphone-array dataset, created following the Sound3DVDet pipeline, report that SoundLoc3D outperforms seven prior methods (mAP 0.518, mAR 0.999, mALE 0.320 versus Sound3DVDet at 0.309, 0.998, 0.586). The paper also includes ablations on view number, microphone number, and loss components, plus robustness tests with acoustic noise and camera-pose noise.","tokens_in":22563,"tokens_out":3697,"duration_ms":33112,"significance":"If the reported results are correct, SoundLoc3D makes a useful empirical contribution to audio-visual 3D sound-source localization in the weakly correlated setting. The method is lightweight (3.8M parameters), simple to describe, and the paper includes several informative ablations plus an explicit release of code and data in the supplementary material. The depth-informed closeness loss and cross-view consistency loss are natural ideas, and the comparison against audio-only baselines and the prior Sound3DVDet system is thorough. However, the evaluation is entirely synthetic, and one per-object baseline table is internally inconsistent with the headline numbers; until that inconsistency is resolved, the claimed state-of-the-art margin over SoundDet cannot be trusted. The robustness claims also cover only certain noise types, not the depth-degradation scenarios that the method's surface assumption would most fear.","major_comments":[{"comment":"The SoundDet row in Table 14 is irreconcilable with the same method's row in Table 2 and per-class Table 13. Table 14 reports SoundDet mAP/mAR/mALE of 0.677/0.815/0.122 on Table, 0.682/0.813/0.110 on Ceiling, and 0.660/0.833/0.108 on Door, whereas Table 2 and Table 13 report overall mAP 0.120, mAR 0.674, mALE 0.823 and per-class AP values around 0.119. The stated standard deviations (at most 0.02) cannot account for a change of about 0.55 in mAP. Since the paper's central claim is a large improvement over SoundDet, this inconsistency must be resolved: either the per-object table is misaligned or erroneous, or the headline comparison is wrong. Please correct and re-verify all tables before resubmission.","section":"Appendix E.3, Table 14; Table 2"},{"comment":"The robustness claim 'robust to RGB-D measurement inaccuracy' is supported only by adding Gaussian noise to camera rotation parameters (Table 6), not by perturbing depth values. The depth-informed closeness loss in Eq. (12) directly assumes the sound source lies on the depth-reconstructed object surface; depth bias, missing-depth regions, or off-surface sources could pull predictions to the wrong surface. Please add experiments with depth noise/bias and missing depth, or restrict the claim to camera-pose inaccuracy.","section":"Section 4.3, Eq. (12)"},{"comment":"The appendix states that the dataset used here is not exactly the same as the original Sound3DVDet dataset because views without depth maps were filtered out. Since all methods are retrained and evaluated on the same filtered set, the comparison is internally fair, but the main text should state this explicitly and should also report the effect of the filtering on Sound3DVDet's performance relative to its original paper. Without this, readers cannot judge how much of the reported gain over Sound3DVDet comes from the depth-modality change versus the dataset re-filtering.","section":"Section 4, Dataset Creation; Appendix C"}],"minor_comments":[{"comment":"The second decoding step writes '(Pupdate,i, Cupdate,i) = D(Qinit,i)' but it should almost certainly be D(Qupdate,i); otherwise the 'updated' queries are never decoded.","section":"Eq. (10)"},{"comment":"The summation is written as 'sum_{j=i}' with the condition i ≠ j; the index should be j ≠ i (or j > i) to avoid a self-pair and to match the stated combination count C = binom(N,2).","section":"Eq. (13)"},{"comment":"The appendix says all standard deviations are within 0.02, while the main text says all are ≤0.010; please make these statements consistent.","section":"Appendix E, Table 14 caption"},{"comment":"The columns in Table 7 are ambiguous: the header 'Num Sound3DVDet Ours' with rows like '5 0.267 0.497' makes it unclear which numbers are mAP values for which method and which are class counts; please reformat.","section":"Table 7"},{"comment":"The definition of Pcentroid,i should state how depth back-projection handles invalid or zero-depth pixels, since the depth-informed loss could otherwise be computed over incomplete data.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical systems paper, but the Table 14/Table 2 discrepancy over SoundDet is exactly the kind of load-bearing internal inconsistency that must be fixed before the headline claim can be accepted. The other major issue is the over-broad robustness statement. I would be willing to accept after these are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a sensible extension of the authors' own Sound3DVDet, adding multiview depth maps to a DETR-style set prediction pipeline, and the simulated numbers look good. But there is a serious internal inconsistency in the appendix that you should chase before trusting the headline claim. Table 2 gives SoundDet mAP/mAR/mALE of 0.120/0.674/0.823, while Table 14 reports SoundDet on Table, Ceiling, and Door as 0.677/0.815/0.122, 0.682/0.813/0.110, and 0.660/0.833/0.108. Those are not within the stated standard deviations; they are irreconcilable. If Table 14 is correct, SoundDet beats SoundLoc3D on those object classes, which contradicts the paper's main result. If it is a copy-paste error, the reporting still needs a fix.\n\nWhat is actually new: the depth-informed closeness loss (Eq. 12) is a clean way to pull predictions toward the object surface, and the cross-view consistency loss is a natural extension of the prior work. The ablations are informative: removing depth hurts more than removing RGB, and the view-number study shows diminishing returns. The method is lightweight (3.8M params) and the robustness tests with pose noise and white noise are a nice touch.\n\nThe soft spots are real but mostly not fatal. The evaluation is entirely synthetic, and the dataset is filtered differently from the original Sound3DVDet, so the comparison with that method is not apples-to-apples. The robustness tests cover additive Gaussian pose noise and white acoustic noise, but not depth bias, missing depth regions, or off-surface sources—exactly the conditions where the depth cue could pull predictions to the wrong surface. There is also no simple geometric baseline (e.g., triangulation from depth projections) that would isolate what the learned components actually add. And the data release being contingent on acceptance is a genuine reproducibility limitation.\n\nBottom line: this deserves a serious referee. The SoundDet numbers in Table 14 must be corrected and explained. If they are a typo, this is a decent incremental contribution; if not, the central claim doesn't hold. I'd ask the authors to fix that before anything else.","headline":"Extension of Sound3DVDet with depth cues shows strong simulated results, but an internal inconsistency in the SoundDet baseline numbers must be resolved before the headline SOTA claim can be trusted.","tokens_in":23084,"tokens_out":3118,"would_cite":false,"duration_ms":25210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding depth maps to a microphone-array camera nearly doubles accuracy for locating invisible sound sources.","keywords":["3D sound source localization","audio-visual learning","microphone array","RGB-D","multiview geometry","set prediction","weakly correlated audio-visual","sound event detection"],"falsifier":"Run the paper's wall-scene dataset with a systematic +0.4 m bias applied to every depth map: if mean localization error rises by roughly 0.4 m, the depth loss is directly pulling predictions onto the biased surface, while if it stays near 0.32 m the loss is more robust than the on-surface assumption would predict.","tokens_in":1770,"feed_emoji":"🎙️","tokens_out":5060,"duration_ms":100051,"temperature":0.7,"pith_summary":"SoundLoc3D aims to establish that invisible 3D sound sources, which cannot be seen but lie on an object's surface, can be reliably localized and classified by combining a four-microphone array with multiview RGB-D images. The paper frames the task as set prediction: each view's audio features generate initial source queries, and multiview RGB-D cues refine those queries to lie on physical surfaces. On a simulated benchmark built from indoor scenes with rendered acoustics, the method reports mAP of 0.518 versus 0.309 for the closest audio-visual prior, with mean localization error dropping from 0.586 m to 0.320 m. A sympathetic reader would care because this addresses a real gap: many fault sounds, like gas leaks or failing fans, are invisible but have a physical location that depth geometry can pin down.","feed_headline":"Depth maps nearly double accuracy for locating invisible sounds","feed_subtitle":"A mic-array camera with multiview depth pins sound sources to object surfaces, beating seven prior methods.","key_machinery":"The load-bearing mechanism is the depth-informed closeness loss (Eq. 12). For each query predicted from one view, SoundLoc3D projects the predicted 3D point into each other view's depth map, reads the depth value along that projection ray, back-projects to get a depth-informed 3D position, and takes the centroid across views. The loss penalizes the L2 distance between the prediction and that centroid once it exceeds 0.3 m, thereby pulling the prediction toward the object surface. This is supported by the cross-view consistency loss (Eq. 13), which ties predictions of the same source across views, and by a pretrained detector-free local feature matcher that supplies multiview appearance features at projected source locations, making visual consistency usable even on texture-homogeneous surfaces.","core_discovery":"The central claim is that depth maps are the key missing modality for weakly correlated audio-visual sound-source localization. SoundLoc3D treats the problem as set prediction: a query generator produces initial sound-source queries from each view's log-mel spectrograms and GCC-PHAT features; a feature mixer then aggregates multiview RGB appearance features at the projected position of each query; and a decoder produces the final 3D position and class. Two losses carry the load: a depth-informed closeness loss that penalizes the distance between the predicted source and the centroid of its multiview depth back-projections beyond 0.3 m, and a cross-view consistency loss that keeps the same source's predictions from different views in agreement. The paper reports that this pipeline outperforms seven prior methods, with mAP 0.518, mAR 0.999, and mALE 0.320, and is robust to added camera-pose noise and white ambient noise. The conclusion is that depth's direct surface geometry, not just RGB appearance, is what makes invisible sources localizable.","pith_inferences":["If the on-surface assumption carries over to real rooms only approximately, a natural extension is a learned or view-dependent tolerance in the depth closeness loss; the paper's fixed 0.3 m threshold is a stand-in for that uncertainty.","The robustness study covers white noise and pose jitter but not biased or missing depth; a test with a systematic depth shift would separate the depth cue's value from its sensitivity to depth error.","The same query-refinement loop could be inverted: audio could disambiguate which of several depth-surfaced objects emitted a sound in occluded scenes, a case the paper does not test."],"forward_implications":["If the reported numbers hold, an acoustic-camera rig can pinpoint invisible sources to an average error of about 0.32 m in simulated indoor scenes, below the 0.5 m true-positive threshold.","Depth contributes more than RGB appearance: removing the depth loss drops mAP from 0.518 to 0.472 and raises mALE to 0.457, while removing RGB keeps mAP at 0.498, so geometry is the primary cross-modal cue.","The method scales from one to six views, with mAP rising from 0.412 to 0.522 and then saturating, meaning only a few viewpoints are needed for most of the benefit.","The framework keeps most of its performance under practical distortions: mAP stays at 0.480 under 0.3 rad camera-pose noise and at 0.467 under 5 dB white noise, while comparators drop further.","Because it treats detection as set prediction, the method handles a variable number of sources without non-maximum suppression and remains stable when the number of sound classes grows from 5 to 9."],"supporting_citations":[{"why":"Provides the closest prior audio-visual baseline, the data-generation strategy, and the multiview RGB pipeline that SoundLoc3D extends with depth.","marker":"[27]"},{"why":"Supplies the pretrained detector-free image matcher whose coarse features give multiview visual consistency at projected source positions.","marker":"[52]"},{"why":"Supplies the acoustic simulator used to render room impulse responses for the synthetic dataset.","marker":"[12]"},{"why":"Supplies the indoor scene dataset that provides the RGB-D geometry in the simulated recordings.","marker":"[10]"},{"why":"Introduces the set-prediction query formulation that SoundLoc3D uses to handle a variable number of sources.","marker":"[9]"},{"why":"Introduces GCC-PHAT, the inter-channel phase feature used to encode spatial cues from the four-microphone array.","marker":"[5]"}],"fun_headline_variants":["Depth maps pinpoint invisible sound sources","Camera depth is the key to locating sounds you can't see","RGB-D acoustic camera pins sound sources to surfaces","Depth is what makes invisible sounds locatable","Multiview depth reveals 3D positions of unseen sounds"],"cache_read_input_tokens":25216,"weakest_assumption_plain":"The depth-informed closeness loss assumes the true sound source lies on the depth-reconstructed object surface; if the source is off-surface, occluded, or the depth is biased, this cue can pull predictions onto the wrong surface.","fun_headline_variants_meta":{"raw":{"variants":["Depth maps pinpoint invisible sound sources","Camera depth is the key to locating sounds you can't see","RGB-D acoustic camera pins sound sources to surfaces","Depth is what makes invisible sounds locatable","Multiview depth reveals 3D positions of unseen sounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001774,"raw_usage":{"total_tokens":7019,"prompt_tokens":993,"completion_tokens":6026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":5953}},"tokens_in":609,"tokens_out":6026,"duration_ms":44140,"temperature":1.0,"reasoning_tokens":5953,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:02:47.118918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's wall-scene dataset with a systematic +0.4 m bias applied to every depth map: if mean localization error rises by roughly 0.4 m, the depth loss is directly pulling predictions onto the biased surface, while if it stays near 0.32 m the loss is more robust than the on-surface assumption would predict.","supporting_citations":[{"cited_title":"Sound3DVDet: 3D Sound Source Detec- tion Using Multiview Microphone Array and RGB Images","cited_arxiv_id":null,"evidence_quote":"Provides the closest prior audio-visual baseline, the data-generation strategy, and the multiview RGB pipeline that SoundLoc3D extends with depth."},{"cited_title":"LoFTR: Detector-Free Local Feature Match- ing with Transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained detector-free image matcher whose coarse features give multiview visual consistency at projected source positions."},{"cited_title":"SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the acoustic simulator used to render room impulse responses for the synthetic dataset."},{"cited_title":"Matterport3D: Learning from RGB- D Data in Indoor Environments","cited_arxiv_id":null,"evidence_quote":"Supplies the indoor scene dataset that provides the RGB-D geometry in the simulated recordings."},{"cited_title":"End-to- end object detection with Transformers","cited_arxiv_id":null,"evidence_quote":"Introduces the set-prediction query formulation that SoundLoc3D uses to handle a variable number of sources."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces GCC-PHAT, the inter-channel phase feature used to encode spatial cues from the four-microphone array."}],"review_version":1}