{"id":"11812167-fd38-40ba-afff-3abeb644741b","arxiv_id":"2507.04845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fusing frozen CLAP and OWL-ViT embeddings via a Cross-Modal Conformer, plus autocorrelation-based features, improves stereo SELD over DCASE 2025 baselines.","lead":"This submission describes audio-only and audio-visual systems for the DCASE 2025 stereo sound event localization and detection task, combining frozen CLAP and OWL-ViT embeddings through a Cross-Modal Conformer architecture. The systems beat the challenge baselines on the development set, with the best ensemble reaching 48.0% F1 versus a 26.8% audio-visual baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Development-set results are selected on the test partition itself, biasing the comparison against fixed baselines; the headline margin may be partly an artifact of this protocol.","rationale":"The paper's headline claim is that the systems substantially outperform the challenge baselines on the development set. The most load-bearing condition for that claim is that the numbers in Table 1 are trustworthy and fairly comparable to the baselines. Section 5.1 explicitly states the best epoch is chosen by F1 on the development test partition, meaning the evaluation partition is used for model selection. This introduces selection bias that is absent from the fixed baselines, so the reported margin is not an apples-to-apples comparison. The reader's weakest assumption (stereo channels having no ITD cues) concerns an input-feature design choice; even if that premise were wrong, the systems could still outperform baselines, and the claim would remain intact. The evaluation-protocol issue directly undermines the quantitative evidence for the central claim. A re-run with a validation split drawn from the training partition would settle whether the observed gains survive unbiased evaluation. Because the reader's verdict is already CONDITIONAL and the proposed check could plausibly confirm the systems still outperform baselines, I do not move the verdict; I only identify a different and more direct threat to the claim than the one the reader highlighted.","tokens_in":9057,"tokens_out":7303,"duration_ms":83812,"concrete_test":"Obtain the training/validation data and code from the referenced Zenodo record. Re-run the final fine-tuning of systems (1)-(4) using a held-out split of the training partition for early stopping (for example, the last 5% of training clips), then compute the ensemble (5) once on the development test partition. If the ensemble F1 drops by more than about 5 points (for example, below roughly 43%) or DOAE rises above about 16 degrees, the reported margin is materially inflated by test-partition-based model selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 5.1, the model is fine-tuned on the training partition of the development set and 'selecting the best based on the highest F1 score on the test partition.' This is a selection-on-the-evaluation-set protocol: the numbers in Table 1 are the best of 80 epochs on the exact partition used for the headline comparison, while the challenge baselines are fixed models evaluated once. The optimism from this protocol means Table 1 does not provide an unbiased estimate of the systems' performance, so the claim of 'substantially outperform' is supported by a biased comparison. The abstract's 'effectiveness of our strategy' is further weakened by the absence of ablations: gains could come from synthetic pretraining, ensembling, or the weighted loss rather than the semantic embedding integration. The stated assumption that stereo channels are FOA-derived and lack ITD (Sec. 3) is a real design concern but is secondary; even if that assumption is false, it would not fix the evaluation bias.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports systems submitted to the audio-only and audio-visual tracks of DCASE 2025 Task 3, which addresses stereo sound event localization and detection in regular video content. The proposed architecture combines a CNN-Conformer SELD encoder with frozen CLAP audio embeddings and OWL-ViT visual embeddings, fused through a Cross-Modal Conformer module. Acoustic input features are log-mel spectrograms, inter-channel level differences, and stpACC autocorrelation features. The models are pre-trained on synthetic audio and audio-visual datasets, fine-tuned on the stereo STARSS23 development set, and further improved through left-right channel swapping, a keypoint-based visual post-processing step, and an ensemble of four systems. On the development set, the submitted systems reach F1 scores of 45.7-48.0% and DOAE of 14.0-15.6 degrees, compared with baseline F1 values of 22.8% (audio-only) and 26.8% (audio-visual) and baseline DOAE values of 24.5 and 23.8 degrees, respectively.","tokens_in":9266,"tokens_out":4768,"duration_ms":53608,"significance":"If the reported margins survive an unbiased evaluation protocol, the paper would demonstrate a meaningful advance in stereo SELD for conventional frontal video, particularly in leveraging large-scale pre-trained language-aligned models to overcome the data-scarcity constraints of multichannel SELD. The synthetic data generation pipeline, the detailed Cross-Modal Conformer design, and the clear reporting of negative results (e.g., keypoint-based DOA correction degrading accuracy) are useful contributions, and the authors provide a public data link. However, the headline comparison is based on model selection on the evaluation partition and lacks ablations, so the evidence for the central claim of superiority is currently weaker than the abstract suggests. The paper is a well-structured technical report, but it needs additional experimental rigor to support a journal-level claim.","major_comments":[{"comment":"The evaluation protocol selects the best epoch by the highest F1 on the test partition of the development set, as stated in Sec. 5.1 ('selecting the best based on the highest F1 score on the test partition'). Table 1 therefore reports the optimistically selected result of an 80-epoch run, while the challenge baselines are fixed models evaluated once. This selection-on-the-evaluation-set protocol biases the comparison in favor of the submitted systems, and no confidence intervals or significance tests are provided. The abstract's claim that the systems 'substantially outperform' the baselines is thus not supported by an unbiased estimate; please report results under validation-based model selection or provide confidence intervals and a paired significance analysis.","section":"Sec. 5.1 / Table 1"},{"comment":"The 10-point RDE improvement over baselines is attributed 'in part to the inclusion of stpACC features', but no ablation removes stpACC or replaces it with a baseline feature. Similarly, the core claim that semantic embeddings from CLAP and OWL-ViT are responsible for the gains is not tested: the submitted systems also differ from the baselines by synthetic pretraining, CMC fusion, the weighted on/off-screen loss, and ensembling. Without ablations, the title and abstract's attribution of the results to spatial and semantic embedding integration is over-claimed. Please add at least a feature ablation (with/without stpACC) and a modality ablation (with/without CLAP and OWL-ViT branches).","section":"Sec. 5.4"},{"comment":"The acoustic feature design rests on the assumption that the stereo channels are 'arithmetically derived from FOA signals' and therefore have no inter-channel time or phase differences, so ITD is discarded and ILD is used as the primary spatial cue. This assumption is load-bearing but unverified. If the stereo generation pipeline introduces even small channel delays, or if the assumption fails for other stereo content, the model would be unable to exploit potentially useful ITD cues. Please provide a quantitative verification, e.g., a cross-correlation analysis of the left and right channels in the development set, or a precise description of the stereo generation code.","section":"Sec. 3"}],"minor_comments":[{"comment":"The sentence 'Tin is chosen so that Tin/16 matches the label frame rate' is unclear; for 5-second clips at 10 labels per second, please state the exact input dimensions and how the temporal pooling aligns with the label rate.","section":"Sec. 2.1"},{"comment":"The phrase 'where m in the mel frequency index' should read 'where m is the mel frequency index.'","section":"Eq. (1)"},{"comment":"The text says the keypoint-based DOA correction resulted in a slight decrease in spatial accuracy, but Table 1 shows no change in DOAE for systems (3.1), (4.1), and (5.1); please clarify whether the decrease was observed on a validation set and is therefore not reflected in the table.","section":"Sec. 5.2"},{"comment":"The ensemble rule contains several hand-set parameters (the 20-degree agreement threshold, the requirement of at least two systems, and the Bell/Knock exceptions); please state whether these were tuned on the development set and, if so, treat them as additional selected hyperparameters.","section":"Sec. 5.3"},{"comment":"The manuscript gives a data link but no code link; a statement on code availability would improve reproducibility, especially for the synthetic data generation and the Cross-Modal Conformer implementation.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"This is a DCASE challenge technical report rather than a full journal paper; the scope and level of experimental detail are appropriate for a challenge report, but the evaluation protocol and missing ablations need to be addressed before the manuscript can support a journal-level claim of superiority. The authors' own concluding statement that future work will investigate the contribution of each modality effectively concedes that the central attribution is not yet established. I do not see evidence of problematic citation or novelty disclosure: the stpACC feature is clearly attributed to the authors' own prior work [19], and the keypoint-based post-processing is credited to Jiang et al. [41]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is credible but the evaluation protocol puts a ceiling on how much we should trust the margin. The systems in this DCASE 2025 report do beat the baselines by a wide gap (F1 48 vs 26.8 on the dev set), but that number is selected on the test partition, so part of the gain is self-fulfilling. That's the first thing to know.\n\nWhat's actually new: the Cross-Modal Conformer (CMC) that adapts Conformer cross-attention for SELD with CLAP audio embeddings and OWL-ViT visual tokens. That fusion is a reasonable engineering step, and preserving OWL-ViT's spatial token layout rather than pooling them away is a sensible choice for a localization task. The synthetic data pipeline (SpatialScaper plus SELDVisualSynth with curated images) is detailed and the data is released, which is genuinely useful for reproducibility. The stpACC feature is from their own prior work; they cite it properly.\n\nThe soft spots are real but not fatal to the main claim. First and most important: Sec 5.1 says they fine-tune on the training partition and select the best epoch based on F1 on the test partition. That's model selection on the evaluation set. It does not make the architecture worthless, but it means Table 1 is not an unbiased estimate of performance. Second, there are no ablations. The abstract says the strategy works, but the headline gain could come from synthetic pretraining, the ensemble, or the weighted loss rather than semantic embeddings. They even attribute the RDE improvement to stpACC in Sec 5.4 without testing that. Third, no error bars or significance tests; with one run per configuration, small differences (e.g., system 1 vs 2) are noise. Minor concern: the assumption that stereo channels have no ITD is stated clearly, but if the dataset actually contains ITD, the ILD-only design would throw away useful cues. That's something an independent check of the data could settle.\n\nOverall, this is a solid challenge report, not a scientific paper. The architecture is worth reading if you work on audio-visual SELD or stereo localization, and the released data is a nice contribution. The evaluation protocol needs fixing before the numbers are used as evidence. I'd send it to review if the authors could either add a validation split for model selection or provide an unbiased comparison with fixed epochs.","headline":"A useful stereo SELD system report with a credible architecture, but model selection on the test partition inflates the headline margin.","tokens_in":9816,"tokens_out":2382,"would_cite":true,"duration_ms":24676,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing frozen CLAP audio and OWL-ViT video embeddings into a Cross-Modal Conformer roughly doubles detection F1 and halves direction error for stereo sound-event localization, versus challenge baselines.","keywords":["Sound Event Localization and Detection","Stereo audio","Cross-Modal Conformer","CLAP","OWL-ViT","Inter-channel level difference","Autocorrelation features","Audio-visual fusion"],"falsifier":"A direct falsifier is to measure the cross-correlation peak delay between the left and right channels on clips with known source azimuth: a systematic nonzero lag would contradict the paper's premise that the stereo channels carry no time or phase differences, and would undermine the ILD-only spatial feature design.","tokens_in":8839,"feed_emoji":"🎧","tokens_out":11267,"duration_ms":103369,"temperature":0.7,"pith_summary":"This paper claims that a major bottleneck in sound event localization and detection (SELD) on ordinary stereo video is semantic: spatial channels say where a sound comes from, but identifying what is making it and whether it is visible is what separates strong systems from weak ones. To test this, the authors fuse frozen, language-aligned semantic embeddings — CLAP for audio and OWL-ViT for video — into a standard CNN-Conformer SELD backbone using a Cross-Modal Conformer that lets one modality attend to another. Pretraining on large synthetic stereo scenes, augmenting by swapping left and right channels, and adding autocorrelation-based features for distance, their systems roughly double detection F1 and cut direction-of-arrival error roughly in half relative to the challenge baselines on the development set. The strongest audio-visual ensemble reaches F1=48.0% and a mean direction-of-arrival error of 14.0°, against an audio-visual baseline of F1=26.8% and 23.8°. If the approach holds, large pretrained semantic models can be plugged into SELD without multichannel microphone arrays, which matters for consumer video and media production.","feed_headline":"Semantic embeddings nearly double stereo sound-localization accuracy","feed_subtitle":"Frozen CLAP and OWL-ViT embeddings lift detection F1 from 26.8% to 48.0% and cut direction error from 23.8° to 14.0°.","key_machinery":"The central mechanism is the Cross-Modal Conformer (CMC), an adaptation of the Conformer (a convolution-augmented transformer) in which two feed-forward modules process two modalities in parallel and multi-head self-attention is replaced by multi-head cross-attention, with queries from one modality and keys and values from the other. The audio branch uses frozen CLAP embeddings as semantic keys and values for the SELD encoder's output, while the video branch uses frozen OWL-ViT patch tokens as keys and values for the combined audio representation, deliberately preserving the spatial layout of image patches rather than average-pooling it away. Stereo spatial cues are carried by inter-channel level differences (ILD) computed from the two channels' STFT magnitudes, alongside per-channel log-mel spectrograms; short-term power of the autocorrelation (stpACC) features, which encode reverberation-related cues, are added to support distance estimation. Training uses the auxiliary duplicating permutation invariant training (ADPIT) loss for up to three tracks, with synthetic FOA scenes generated by SpatialScaper, synthetic videos from SELDVisualSynth, and left-right audio channel swapping plus video flipping as augmentation.","core_discovery":"The paper's central claim is that semantic information from contrastive language-aligned models can be injected into a standard CNN-Conformer SELD backbone through a Cross-Modal Conformer, and that this combination — together with synthetic pre-training, left-right channel and video swap augmentation, and autocorrelation-based distance features — yields the reported gains on stereo SELD in regular video. On the development set, the audio-only systems reach F1 scores of 45.7% and 46.0% with direction-of-arrival errors of 15.0° and 15.2°, compared with an audio-only baseline of F1=22.8% and DOAE=24.5°. The audio-visual ensemble reaches F1=48.0%, DOAE=14.0°, on/off-screen accuracy of 80.8%, and a relative distance error of 29.3%, compared with an audio-visual baseline of F1=26.8%, DOAE=23.8%, and RDE=40.0%. The paper further claims that the roughly ten-percentage-point improvement in relative distance error is partly due to the stpACC autocorrelation features, and that human-keypoint post-processing improves only on/off-screen classification, not localization.","pith_inferences":["An implication the report leaves implicit is that the same cross-modal fusion recipe should transfer to other low-channel-count settings, such as monophonic consumer video or binaural recordings, where frozen semantic embeddings could compensate for reduced spatial information; this is testable by replacing the stereo front-end while keeping the CMC and encoders fixed.","Because the paper does not ablate CLAP, OWL-ViT, and stpACC separately, the relative contribution of each component is unknown; a natural follow-up is to remove each branch and measure F1, DOAE, and RDE on the same development partition.","The keypoint post-processing result suggests visual semantics are more reliably used as a prior for what is visible than as a correction to acoustic localization; one could test this by feeding keypoint-object association scores into the Cross-Modal Conformer rather than applying them after the fact.","Since the stereo channels are arithmetically derived from FOA, a system built on ILD may not generalize to true binaural recordings with physical microphone spacing; a domain-transfer experiment on binaural data would reveal how much of the gain depends on the derived-stereo premise."],"forward_implications":["A SELD system can use frozen, pre-trained contrastive language-aligned encoders as a source of semantic information, avoiding the need to train those encoders on multichannel audio.","On stereo-only video input, both the audio-only and audio-visual systems roughly double F1 and reduce direction-of-arrival error by about ten degrees relative to the challenge baselines on the development set.","Combining four systems by an agreement-based ensemble, with per-class exceptions for Bell and Knock, adds further gains: the ensemble reaches F1=48.0% and DOAE=14.0°.","Autocorrelation-based stpACC features are credited with part of the roughly ten-percentage-point improvement in relative distance error.","Human-keypoint post-processing did not improve localization and was used only to correct on/off-screen labels, indicating a boundary of visual semantic refinement."],"supporting_citations":[{"why":"Supplies the stereo STARSS23 development set used for fine-tuning and evaluation, defining the task, baselines, and on/off-screen subtask.","marker":"[8]"},{"why":"Provides the frozen CLAP audio encoder whose language-aligned embeddings carry semantic information into the first Cross-Modal Conformer.","marker":"[13]"},{"why":"Provides the frozen OWL-ViT visual encoder whose patch tokens preserve spatial layout and serve as keys and values in the second Cross-Modal Conformer.","marker":"[14]"},{"why":"Introduces the left-right audio channel swapping (ACS) augmentation used to double the stereo training data.","marker":"[15]"},{"why":"Defines the Conformer architecture that the Cross-Modal Conformer adapts, including feed-forward, convolution, and attention module structure.","marker":"[18]"},{"why":"Proposes the short-term power of the autocorrelation (stpACC) features that the paper credits with improving distance estimation.","marker":"[19]"},{"why":"Define the auxiliary duplicating permutation invariant training (ADPIT) loss used to train the model for up to three overlapping tracks.","marker":"[21, 22, 23]"},{"why":"Provides the SpatialScaper library used to generate the synthetic FOA audio clips for pre-training.","marker":"[28]"},{"why":"Provides the SELDVisualSynth tool used to synthesize the corresponding videos for the audio-visual pre-training dataset.","marker":"[36]"}],"fun_headline_variants":["Semantic embeddings lift stereo SELD F1 from 27% to 48%","Cross-Modal Conformer doubles sound-localization accuracy","CLAP and OWL-ViT slash direction error by 41%","Semantic fusion nearly doubles stereo sound-event F1","Pre-trained embeddings boost SELD to 48% F1 on DCASE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the stereo channels in the dataset are arithmetically derived from first-order ambisonics and therefore contain no inter-channel time or phase differences, so the model can safely rely on inter-channel level differences as its only spatial cue.","fun_headline_variants_meta":{"raw":{"variants":["Semantic embeddings lift stereo SELD F1 from 27% to 48%","Cross-Modal Conformer doubles sound-localization accuracy","CLAP and OWL-ViT slash direction error by 41%","Semantic fusion nearly doubles stereo sound-event F1","Pre-trained embeddings boost SELD to 48% F1 on DCASE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3144,"prompt_tokens":1078,"completion_tokens":2066,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":1973}},"tokens_in":694,"tokens_out":2066,"duration_ms":16039,"temperature":1.0,"reasoning_tokens":1973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:37:11.201254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is to measure the cross-correlation peak delay between the left and right channels on clips with known source azimuth: a systematic nonzero lag would contradict the paper's premise that the stereo channels carry no time or phase differences, and would undermine the ILD-only spatial feature design.","supporting_citations":[{"cited_title":"Sound event localization and detection of over- lapping sources using convolutional recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the stereo STARSS23 development set used for fine-tuning and evaluation, defining the task, baselines, and on/off-screen subtask."},{"cited_title":"A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLAP audio encoder whose language-aligned embeddings carry semantic information into the first Cross-Modal Conformer."},{"cited_title":"Simple open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen OWL-ViT visual encoder whose patch tokens preserve spatial layout and serve as keys and values in the second Cross-Modal Conformer."},{"cited_title":"STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,","cited_arxiv_id":null,"evidence_quote":"Introduces the left-right audio channel swapping (ACS) augmentation used to double the stereo training data."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Defines the Conformer architecture that the Cross-Modal Conformer adapts, including feed-forward, convolution, and attention module structure."},{"cited_title":"AudioGPT: understanding and generating speech, music, sound, and talking head,","cited_arxiv_id":null,"evidence_quote":"Proposes the short-term power of the autocorrelation (stpACC) features that the paper credits with improving distance estimation."},{"cited_title":"Sound event detection and localization with dis- tance estimation,","cited_arxiv_id":null,"evidence_quote":"Provides the SpatialScaper library used to generate the synthetic FOA audio clips for pre-training."},{"cited_title":"FSD50K: an open dataset of human-labeled sound events,","cited_arxiv_id":null,"evidence_quote":"Provides the SELDVisualSynth tool used to synthesize the corresponding videos for the audio-visual pre-training dataset."}],"review_version":1}