{"id":"20a0fe47-2f75-4dbf-aa69-5e0c66a9ce61","arxiv_id":"2507.07384","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-band semantic-spatial alignment network (AV-SSAN) localizes a target sound source using a cross-instance visual prompt, achieving 16.59 degrees mean error and 71.29% accuracy on the new VGGSound-SSL benchmark.","lead":"This paper introduces a new task in which a computer finds a sound source, like a barking dog, using a picture of a different dog from the same category. The authors build a synthetic audio-visual dataset and a neural network that combines sound analysis with the picture to estimate the direction of the sound.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VGGSound-SSL benchmark is purely synthetic, and the paper never tests synthetic-to-real transfer: the STARSS23 result trains on real data, not transfers from the synthetic dataset. The claimed real-world utility of the benchmark is therefore unverified.","rationale":"The reader identified synthetic RIR realism as the weakest assumption, and I agree that this is the most load-bearing point. However, the reader's framing says the STARSS23 result is 'the only real-world check' and 'partially validates' the synthetic data. The paper's experimental section states that models are 'further trained' on STARSS23, meaning the real-world evaluation does not actually test transfer from the synthetic VGGSound-SSL training distribution. Thus the concern is stronger than the reader stated: there is currently no evidence that models trained on VGGSound-SSL generalize to real acoustic environments. Additionally, the unspecified microphone array geometry is a concrete, reproducible detail missing from the dataset construction, and it directly affects whether IPD/ILD features are physically meaningful. Because the reader's verdict of CONDITIONAL already asks for additional validation, my analysis does not change the verdict; it sharpens the specific condition that should be added. I would keep the verdict CONDITIONAL, requiring either a zero-shot synthetic-to-real transfer experiment or a clear statement that the benchmark is intended only as a within-synthetic comparison, not as a proxy for real-world deployment. The paper has no fatal internal inconsistencies in the architecture or loss definitions, and the ablation study is informative, but the external validity of the main benchmark is the key unresolved issue.","tokens_in":11244,"tokens_out":11580,"duration_ms":133120,"concrete_test":"Train AV-SSAN on VGGSound-SSL only, and evaluate it zero-shot on the STARSS23 test fold without any fine-tuning on STARSS23. Report MAE/ACC (and, ideally, a small grid over random RIR seeds). If zero-shot MAE is close to the in-domain STARSS23 result (~27°), the synthetic training transfers and the concern is dismissed. If it degrades sharply (e.g., >45°), the synthetic spatial cues are not representative of real rooms, and the benchmark's validity for real-world CI-AVL is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central benchmark for the headline claim is VGGSound-SSL, constructed by convolving single-channel VGGSound audio with 10,000 GPU-RIR room impulse responses. All results in Table 2 are on this synthetic data. The only real-world evaluation, STARSS23 (Table 3), is described as 'further train models on the STARSS23 dataset' — the model is trained or fine-tuned on real recordings, so it does not test whether spatial cues learned from synthetic RIRs transfer to real rooms. Moreover, the microphone array geometry used to produce the two-channel VGGSound-SSL audio is never specified; IPD/ILD are only meaningful relative to a physical array, and the RIR simulation's realism for that array is unvalidated. Consequently, the paper's claim that VGGSound-SSL is a valuable benchmark for CI-AVL, and the implicit claim that the synthetic performance reflects real-world localization ability, rest on an untested assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, Cross-Instance Audio-Visual Localization (CI-AVL), in which the direction-of-arrival (DoA) of a target sound is estimated from two-channel audio given a visual prompt from a different instance of the same semantic class. The authors construct VGGSound-SSL, a synthetic dataset of 13,981 two-channel clips created by convolving VGGSound audio with GPU-RIR room impulse responses, and propose AV-SSAN, a model built around a Multi-Band Semantic-Spatial Alignment Network (MB-SSA Net) that decomposes the spectrogram into fine, mid, and full bands, aligns each band with the visual prompt, and refines the spatial cues before DoA prediction. Experiments on VGGSound-SSL at three SNRs and on STARSS23 report that AV-SSAN outperforms eight audio-only and audio-visual baselines, and ablations show monotonic gains from each component.","tokens_in":11388,"tokens_out":9881,"duration_ms":102754,"significance":"If the results hold, the paper contributes a useful task formulation and a plausible architecture. CI-AVL relaxes the spatial-pairing requirement of prior audio-visual sound source localization work, and the explicit multi-band semantic-spatial alignment is an interesting design that is supported by the ablation study. The dataset, if properly validated, could become a large-scale benchmark for prompt-based DoA estimation. However, the current evidence is not yet sufficient: the synthetic benchmark's acoustic realism is unvalidated, the reported gains lack statistical support, several central configuration choices are undocumented, and the comparison set omits the most closely related selective-localization methods.","major_comments":[{"comment":"The array geometry used with GPU-RIR is never specified: the reader is not told the number and spacing of microphones, their orientation, or the source-receiver configuration that produces the two-channel inputs. Since the model's primary cues are IPD and ILD (Eqs. (2)-(3)), which are meaningful only relative to a physical array, this omission blocks reproduction and prevents any check that the simulated spatial cues resemble real recordings. Please report the geometry and include a validation of the RIR simulation against measured array data or a real-recorded benchmark.","section":"Dataset Construction (Spatial Audio Synthesis)"},{"comment":"All reported numbers come from a single training run with no error bars, confidence intervals, or significance tests. The word 'significantly' in the abstract is therefore not supported, and several margins are small: at 0 dB, CMAF (18.65) and AV-SELD (18.95) are close, while on STARSS23 AV-SELD (28.54) and AV-SSAN (27.46) differ by about 1 degree. Please report means and standard deviations over at least three seeds and, where possible, paired significance tests between the best baseline and AV-SSAN.","section":"Training and Evaluation; Tables 2 and 3"},{"comment":"The target distribution p(theta) is written as exp(-|theta_hat - theta|/sigma_theta), but theta_hat is used both as the discretized DoA variable and later as the predicted class, so as written the training target depends on the model's own prediction. Please use distinct notation for the ground-truth class and the predicted class, and report the value of sigma_theta, which is never given and controls the width of the target distribution and hence the scale of L_DoA.","section":"Problem Formulation, Eq. (13)"},{"comment":"The STARSS23 experiment does not test synthetic-to-real transfer: the text says models are additionally trained on STARSS23, so Table 3 measures performance after training or fine-tuning on real data, not whether cues learned on VGGSound-SSL generalize. The paragraph calling this 'transferability' is misleading. A direct evaluation of a VGGSound-SSL-trained model on STARSS23 would address this; otherwise the wording should be revised and the limitation stated.","section":"Training and Evaluation; Table 3"},{"comment":"The closest prior work in the selective-localization setting (class-conditioned SELD, text-queried SSL, and LocSelect) is discussed in Related Work but never evaluated. Since CI-AVL is a selective localization task, comparisons only against non-selective AV-SSL baselines do not establish the advantage of the proposed approach over methods designed for target-conditioned localization. Please add at least one such comparison, or justify why these methods cannot be adapted to visual prompts.","section":"Baselines; Training and Evaluation"},{"comment":"The ground-truth spectrogram X_gt used in the reconstruction loss is not defined. If X is the mixed spectrogram and X_gt is the target-only spectrogram, then the loss trains a separation mask and this should be stated; if X_gt equals the input, the loss is a trivial identity objective. Please define X_gt precisely and specify whether the mask is supervised by the target source's spectrogram.","section":"Training Loss, Eq. (12)"}],"minor_comments":[{"comment":"The dataset name is inconsistent: Table 2's caption and the conclusion call it 'VGG-SSL' while the rest of the paper calls it 'VGGSound-SSL'; please standardize.","section":"Throughout"},{"comment":"The phrase 'tested under SNRs ranging from -5 dB to -10 dB' should read 'at -5 dB and -10 dB,' since training is at 0 dB.","section":"Experiments and Discussions"},{"comment":"p(theta) is described as 'Gaussian-like' but exp(-|x|/sigma) is a Laplace distribution; please correct the terminology.","section":"Problem Formulation"},{"comment":"A_b^i has matrix form but is then treated as a vector when concatenated and pooled; please specify the exact dimensions and the pooling operation used to obtain the scalar beta_b^i.","section":"MB-SSA Net, Eqs. (6)-(7)"},{"comment":"The color/weight encoding ('Larger Weights', 'Lower Weights') in the figure is not explained in the caption; please add a legend or explicit caption text.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an audio/speech signal processing venue. The main risks are empirical: the synthetic benchmark is unvalidated and the statistical evidence is thin. I did not identify any obvious novelty or integrity concern. If the manuscript is considered further, I would encourage the editor to make code and data availability a condition, since the paper promises public release upon acceptance and several details (array geometry, sigma_theta, baseline adaptations) are currently only promised in appendices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jian,\n\nThe paper is worth a look, but the headline result is less solid than it appears. The genuinely new thing here is the CI-AVL task — localizing a sound source from a visual prompt of a different instance of the same class — and the VGGSound-SSL dataset built for it. That framing is useful and I don't know a prior paper that does exactly this. The architecture is a reasonable combination of known pieces: CLIP+VGGish for semantics, tri-band IPD/ILD for spatial features, cross-attention, and a mask-based refiner. The ablations show each component adds something, and the qualitative examples are consistent with the story.\n\nThe soft spots are real, though. The biggest one: the paper claims VGGSound-SSL is a valuable benchmark, but its realism is never tested. The dataset is made by convolving VGGSound audio with GPU-RIR impulses. All the headline numbers in Table 2 come from that synthetic data. The STARSS23 experiment does not validate the synthetic cues, because the models are retrained on STARSS23 instead of transferring from the synthetic set. The text even calls it a 'transferability' check, which overstates what was done. On top of that, the microphone array geometry used to generate the two-channel audio isn't given in the main text, so IPD/ILD are not reproducible from the description.\n\nThere are other issues, mostly in the minor-to-moderate range. The tables have no error bars or significance tests; everything looks like a single run. The baselines are AV-SSL methods that expect spatially-paired video frames, and they're fed cross-instance prompts they weren't designed for, so the comparison is not entirely clean — though it's a reasonable first benchmark. The sigma_theta in the DoA target distribution isn't reported, which matters because the loss is an MSE between predicted and target distributions.\n\nThe citation pattern looks fine, and the work is clearly presented. No circular argument or fabricated numbers. The central assumption — that synthetic RIRs give realistic spatial cues — is plausible but unverified.\n\nWho is this for? People working on sound event localization, selective auditory attention, or audio-visual perception will want to know about the task and dataset. It deserves a serious referee, but the authors should be pushed to either demonstrate synthetic-to-real transfer (train on VGGSound-SSL, test on STARSS23 or another real dataset) or soften the claim about the benchmark, and to add variance estimates and release the code and data.\n\nMy take: send it to review, but flag the transfer experiment as the key revision.","headline":"Genuinely new task and dataset, but the synthetic benchmark's realism is untested and the STARSS23 result doesn't demonstrate transfer; worth refereeing with revisions.","tokens_in":11973,"tokens_out":3086,"would_cite":true,"duration_ms":35618,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that a sound source can be selectively localized from a photo of a different instance of the same class, via explicit semantic-then-spatial alignment, and reports top results on a new benchmark.","keywords":["audio-visual sound source localization","direction-of-arrival estimation","cross-instance visual prompting","semantic-spatial alignment","multi-band frequency decomposition","selective localization","VGGSound-SSL dataset"],"falsifier":"Train AV-SSAN only on synthetic VGGSound-SSL, then record a two-microphone array in a real room with a known source position and a same-class prompt image; if the model's MAE on these real recordings is no better than the audio-only baselines, or its accuracy approaches chance, the synthetic spatialization premise fails.","tokens_in":10989,"feed_emoji":"🎯","tokens_out":6292,"duration_ms":67163,"temperature":0.7,"pith_summary":"This paper introduces a task, Cross-Instance Audio-Visual Localization, in which the goal is to estimate the direction of a target sound using only a photo of a different object of the same category, with no requirement that the photo and the sound come from the same recording. It argues that existing audio-visual localization methods fail this task because they align audio and video spatially ('where') without first establishing semantic identity ('what'). To solve it, the paper proposes AV-SSAN, which builds a semantic bridge between the prompt and the audio, then aligns each frequency band of the spatial audio with that semantics and refines the band-wise cues to predict direction of arrival. On the newly constructed VGGSound-SSL benchmark, AV-SSAN reports 16.59 degrees mean absolute error and 71.29 percent accuracy at 0 dB SNR, and it also reports the best transfer result on the real-world STARSS23 set. If these results hold, selective, prompt-driven localization becomes practical without spatially paired audio-visual data.","feed_headline":"A same-class photo steers sound localization to 16.59° error","feed_subtitle":"Cross-instance visual prompts let AV-SSAN isolate one sound in a mixture, beating baselines on synthetic and real recordings.","key_machinery":"The central object is the Multi-Band Semantic-Spatial Alignment Network (MB-SSA Net), a module that couples semantic identity with frequency-dependent spatial cues. It splits each spectrogram into three resolutions—fine bands of 32-bin width, mid bands of 128-bin width, and the full spectrum—and computes interaural phase difference (IPD) and interaural level difference (ILD) per band. Each band passes through a Semantic-Spatial Alignment (SSA) block in which the semantic bridge feature (built from the visual prompt and semantic audio embedding) acts as the query in cross-attention over the band's spatial features; an attention map from the same query-key product yields scalar band-importance weights that aggregate the patches. A spatial refiner then predicts temporal weights across bands and a time-frequency mask that is applied back to the original spectrogram before the DoA predictor. The design carries the argument because it lets the network suppress interference-dominated frequencies and keep target-dominated ones, which the ablations connect to most of the accuracy gain.","core_discovery":"The central claim is that selective sound localization can be driven by semantic, not spatial, correspondence: a visual prompt from a different instance of the same class is enough to isolate and localize the matching source inside a mixture of two sounds. The authors implement this as an explicit two-stage pipeline: first, a semantic bridge fuses the prompt's visual embedding with a semantic audio embedding; second, the Multi-Band Semantic-Spatial Alignment Network decomposes the spectrogram into fine, mid, and full frequency bands, computes interaural phase and level differences, and uses the semantic bridge as a query in cross-attention so each band selects target-related spatial evidence. A semantic-guided band fusion weights the bands and a spatial refiner applies the resulting mask to the original spectrogram before DoA prediction. On the synthetic VGGSound-SSL test at 0 dB SNR the framework reaches 16.59 degrees MAE and 71.29 percent accuracy, and on STARSS23 it reaches 27.46 degrees MAE and 52.31 percent accuracy, both ahead of the audio-only and audio-visual baselines compared in the paper.","pith_inferences":["Extension beyond the paper: the learned band-attention weights could be read as an explanation of which frequency region carries the target, turning the model into a diagnostic for acoustic cue use, though the paper does not evaluate this interpretation.","Extension beyond the paper: if synthetic spatialization transfers as claimed, large video corpora can be converted into spatial-audio training data without recording arrays, which would make DoA training scalable to far more sound classes than current real-world datasets.","Extension beyond the paper: the paper leaves open how performance degrades when the prompt image is weakly representative of the target class; a stress test with noisy or atypical same-class prompts would clarify the limits of the semantic-spatial alignment.","Extension beyond the paper: the two-stage what-then-where design may apply to other multimodal localization problems, such as tracking a speaker named by a text caption or by a reference voice, where the visual reference is not spatially aligned with the audio."],"forward_implications":["A user can select which sounding object to locate using only a photo of a different object of the same category, so applications like robotic listening, hearing-assist devices, or video surveillance can query a target without needing the exact paired recording.","Existing audio-visual localization methods, which fuse spatially aligned audio and video, are outperformed by the semantic alignment route on the new benchmark, suggesting that semantic identity should be disentangled from spatial fusion.","Decomposing spatial cues by frequency band and weighting bands by semantic relevance yields gains beyond full-band fusion; the ablation attributes about three percentage points of accuracy to multi-band modeling and band attention.","The VGGSound-SSL dataset (13,981 clips, 296 classes) provides a shared benchmark for cross-instance audio-visual localization, and the reported transfer to STARSS23 indicates the method behaves on real recordings.","Because the visual prompt enters through a semantic embedding rather than a spatially paired frame, the same design could in principle accept other semantic query modalities, such as text descriptions or reference audio, extending selective localization to categories without any image."],"supporting_citations":[{"why":"Supplies the VGGSound video-audio collection from which the 13,981 spatial clips and prompt images are derived.","marker":"(Chen et al. 2020)"},{"why":"Provides the GPU-accelerated room impulse response synthesis used to spatialize single-channel audio into two-channel clips.","marker":"(Diaz-Guerra, Miguel, and Beltran 2021)"},{"why":"Provides the pretrained cross-modal encoder used to embed prompt images and select representative frames by image-text similarity.","marker":"(Radford et al. 2021)"},{"why":"Provides the VGGish audio embedding network used to build the semantic audio representation in the semantic bridge.","marker":"(Hershey et al. 2017)"},{"why":"Supplies STARSS23, the real-world spatial recording benchmark used to test transferability beyond synthetic data.","marker":"(Shimada et al. 2023)"},{"why":"Defines an audio-visual DoA baseline (MLP-AVC) that the method must beat on both the synthetic and real benchmarks.","marker":"(Qian et al. 2021b)"}],"fun_headline_variants":["A photo of the same class isolates one sound in a mixture","Semantic cue picks a sound source without spatial audio pairing","Cross-instance prompts beat audio-visual baselines on DoA","Multiband alignment with class photo yields 16.59° MAE","Selective sound localization from a same-class visual prompt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that the synthetic room impulse responses used to spatialize VGGSound audio produce interaural cues and ground-truth directions representative enough of real rooms that a model trained on them will localize in the wild; if that premise fails, the VGGSound-SSL results overstate real localization ability and STARSS23 becomes the only test.","fun_headline_variants_meta":{"raw":{"variants":["A photo of the same class isolates one sound in a mixture","Semantic cue picks a sound source without spatial audio pairing","Cross-instance prompts beat audio-visual baselines on DoA","Multiband alignment with class photo yields 16.59° MAE","Selective sound localization from a same-class visual prompt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1535,"prompt_tokens":1009,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":625,"tokens_out":526,"duration_ms":5580,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:42:33.095979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train AV-SSAN only on synthetic VGGSound-SSL, then record a two-microphone array in a real room with a known source position and a same-class prompt image; if the model's MAE on these real recordings is no better than the audio-only baselines, or its accuracy approaches chance, the synthetic spatialization premise fails.","supporting_citations":[],"review_version":1}