REVIEW 4 major objections 6 minor 32 references
A Two-Step Learning Framework for Enhancing Sound Event Localization and Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that separating detection and direction training, then fusing features, cuts the 2023 baseline SELD score by 18.8 percent.
desk verdict A sensible SELD system paper with a reusable trackwise reordering idea, but the headline score is one run and the margins are likely within training variance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a two-step pipeline built around three components. First, the DoA network (based on IPDnet) is modified with spectral convolution in the full-band path and multi-head self-attention with positional encoding in the narrow-band path, and it regresses Cartesian coordinates for up to K overlapping tracks. Second, an ATST-SED feature extractor is extended to a trackwise output with $n_{\text{class}}\times n_{\text{tracks}}$ logits, and a trackwise reordering format keeps each event type in a fixed track across the clip so that per-track steering vectors are stable. Third, a delay-and-sum beamformer uses the predicted coordinates to compute per-microphone distances and steering vectors (Eqs. (1)--(4)), gates inactive tracks with a $0.5$ m threshold, and feeds the beamformed output together with DoA and SED features into a fusion stage built from a Spatial-Acoustic Attention Module and Feature Interaction Gate.
What would settle it
Retrain the system with the track-weighting threshold from $0.1$ m to $2.0$ m, or inject angular noise of roughly $10^\circ$--$20^\circ$ into the predicted coordinates used in Eqs. (1)--(4); if the SELD score barely moves, the beamforming stage is not carrying the benefit the paper attributes to it.
Extended reading notes
Core claim
The central claim is that the two-step learning framework outperforms the 2023 baseline across every evaluation metric, reducing the composite SELD score by 18.8 percent and reaching 0.3891 on the development test set. The paper attributes this gain to three coordinated changes: training the direction-of-arrival (DoA) and sound-event detection (SED) networks independently before any joint fine-tuning; a trackwise reordering format that stops event labels from switching tracks across time; and a fusion stage in which delay-and-sum beamformed features are combined with DoA and SED feature streams through attention-based gates. The strongest stated result is that this combination also improves on recent single-branch and dual-branch systems, with a 3.2 percent SELD-score reduction over CST-former and 2.2 percent over MFF-EINV2 on the same development test set.
Load-bearing premise
The load-bearing premise is that the first-step direction-of-arrival predictions are accurate enough to compute steering vectors, and that the fixed $0.5$ m distance threshold in the track weighting correctly separates active from inactive tracks; the paper reports no sensitivity analysis for either.
Editorial extensions
If this is right
- If the two-step recipe works as claimed, SELD systems can be improved by reusing existing, individually strong SED and DoA models rather than redesigning a joint architecture.
- The trackwise reordering format should prevent the same event type from being reassigned across tracks over time, which should make downstream tracking and beamforming more stable.
- The beamformed spatial features should let the fused network exploit source-direction information even when the event-classification branch is uncertain, and vice versa.
- The reported gains over recent single-branch and dual-branch baselines suggest that task separation followed by fusion is a viable alternative to shared-parameter architectures.
- The three ablation configurations (+DoA fusion, +BF, +SED fusion) show that each added feature stream contributes monotonically to the SELD score in the reported setting.
Reading between the lines
- One implication the authors leave implicit: the same two-step recipe could transfer to other paired spatial-audio tasks, such as speech separation with location guidance, by swapping the event network for the target classifier.
- Because the trackwise reordering and the beamforming are introduced together in the reported experiments, the framework does not isolate which one drives the 18.8 percent gain; a factorial ablation would separate the contributions.
- The hand-set 0.5 m threshold is a calibration liability in real deployments; a learned distance-confidence gate would make the beamforming step self-adaptive.
- If the DoA network's coordinate errors are biased rather than random, the steering vectors could systematically suppress the true source; testing with synthetic impulse responses of known geometry would expose this failure mode.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-step learning framework for sound event localization and detection (SELD). In the first step, a DoA network (a modified IPDnet with spectral convolution and temporal multi-head self-attention) and an SED network (ATST-SED extended with a trackwise output format) are trained separately. In the second step, the DoA, beamformed, and SED features are fused via a LA-VT-style attention module with SAAM and FIG blocks. On the DCASE 2023 Task 3 development test set, the full system reports a SELD score of 0.3891, which the paper claims is the lowest among the compared methods, with an 18.8% relative reduction over the 2023 baseline and 2.2%/3.2% improvements over MFF-EINV2 and CST-former, respectively.
Significance. If the reported score is reliable, the paper would demonstrate a useful combination of task-specific pretrained networks, a trackwise reordering label format, and spatial-acoustic feature fusion for SELD. The trackwise reordering idea and the two-step fusion strategy are of interest to the SELD community and are supported by internal ablations in the right direction. However, the evidence as presented is thinner than the claims require: the margins over prior systems are within typical run-to-run variance for this task, the beamforming gate rests on an uncalibrated distance threshold, and no code or checkpoints are provided. The paper ships no machine-checked proofs or reproducible artifacts; its value is entirely empirical, so the statistical and reproducibility gaps are load-bearing rather than cosmetic.
major comments (4)
- [§3.2.3, Table 3] The headline claim of a SELD score of 0.3891 and the 2.2–3.2% relative improvements over MFF-EINV2 and CST-former rest on a single training run with no seeds, confidence intervals, or significance tests. The ablation increments between +DoA fusion (0.4007), +BF (0.3966), and +SED fusion (0.3891) are 0.0041 and 0.0075 in SELD score, which is the same order as typical retraining variance for end-to-end SELD systems. Please report multi-seed mean and standard deviation, or paired significance tests, for at least the three fusion configurations and the main baselines; otherwise, the component gains and the superiority over prior methods are not identifiable from the numbers presented.
- [§2.2.1, Eq. (3)] The beamforming gate uses a hand-set distance threshold of 0.5 m applied to d_k(t), but the DCASE 2023 Task 3 dataset provides directions, not source distances, and the DoA network predicts Cartesian coordinates whose scale is not defined. The Euclidean distance in Eq. (1) is therefore an arbitrary unit-sphere convention rather than a physical distance, and the threshold in Eq. (3) is not calibrated to any measured quantity. No sensitivity analysis is given, and Table 1 reports MAE of 17.6° against a 20° acceptance threshold, so biased coordinate estimates could make the gate unstable and the beamformed features harmful. Please either calibrate or remove this threshold, report SELD score as a function of the threshold value, and clarify what physical units, if any, the predicted coordinates carry.
- [§2.1.3] The trackwise reordering format is presented as a central contribution, but the manuscript provides only a qualitative example and does not specify the assignment or reordering algorithm: how tracks are matched across frames, how new tracks are initialized, how the maximum track count K is enforced, or how ties are broken. Without a precise procedure, the SED and DoA track outputs cannot be reproduced, and the claimed temporal-consistency benefit is not directly measured. Please provide a formal algorithm or pseudo-code and an evaluation of track-switch rates with and without the proposed reordering.
- [§3.2.3 and §3.1] All baseline scores are taken from prior publications rather than re-run under the paper's evaluation protocol, and no code or checkpoints are released. Because the proposed system uses different pretrained extractors, batch sizes, learning-rate schedules, and preprocessing (Section 3.1), the reported comparison may reflect implementation differences rather than an architectural advantage. Please either re-run the closest baselines under the same harness or release code and checkpoints so that the single reported number can be independently verified.
minor comments (6)
- [Abstract] The word "tracwise" in the abstract is a typo and should be "trackwise."
- [Table 2 caption] The caption contains the typo "segemnt" for "segment."
- [§3.2.3] The sentences containing "confims" and "effectivess" contain typos and should read "confirms" and "effectiveness."
- [Eq. (3)] The word "defind" is a typo and should be "defined."
- [Figure 2] The notation in Figure 2 is confusing: the caption says inputs are formatted as "batch size × (dimensions of a single sample within the batch)," but the figure itself does not show a batch dimension, and M, T, F are not defined in the caption.
- [§3.2.1] Table 1 would be easier to interpret if the evaluation subset and metric definitions were stated explicitly, since the DoA network is evaluated separately from the final SELD system.
Circularity Check
No significant circularity: the two-step SELD framework is evaluated against external DCASE 2023 Task 3 baselines, and the only self-citation is a provenance note, not a load-bearing premise.
full rationale
The paper's derivation chain is empirical rather than definitional. Step 1 trains a DoA network (IPDnet with spectral-convolution and MHSA modifications) and an SED network (ATST-SED with trackwise output) using standard supervised losses (MSE, BCE) against STARSS23 labels. Step 2 forms delay-and-sum beams from the frozen DoA predictions (Eqs. 1-4) and fuses DoA/SED features via SAAM/FIG; the beamformer is a feature-generation stage, not a target quantity defined in terms of itself. The final SELD score is compared with published baselines (CST-former, MFF-EINV2, DST attention, DCASE baseline) on the same development test set, so the headline comparison is externally anchored. The trackwise reordering format is an output-label convention; Table 2 evaluates it by converting to non-trackwise with max pooling, and the improvement is not forced by construction. No fitted parameter is renamed as a prediction, and no uniqueness result is imported from the authors' prior work. The only self-citation, [15], acknowledges prior DCASE work but is not used as evidence for the numerical claims; those claims rest on Tables 1-3. Correctness risks such as single-run variance and the hand-set beamforming threshold are empirical concerns, not circularity.
Assumptions & free parameters
free parameters (1)
- Beamforming distance threshold =
0.5
assumptions (4)
- domain assumption The maximum number of overlapping sound events is K=6, matching the dataset's track count.
- standard math STFT and log-mel processing with a hop length of 0.02 s and window size of 0.04 s captures sufficient spectral and temporal detail for the tasks.
- domain assumption ATST-SED's pretrained feature extractor, frozen during training, transfers useful representations to the DCASE 2023 Task 3 subset used here.
- domain assumption Near-field model: source coordinates can be used to compute distances to each microphone and thus steering vectors (Eqs. 1-2).
Cite this review
Pith. "Pith review of A Two-Step Learning Framework for Enhancing Sound Event Localization and Detection." pith.science (2026). https://pith.science/paper/K2EENXT5
@misc{pith2026250722322,
author = {Pith},
title = {Pith review of: A Two-Step Learning Framework for Enhancing Sound Event Localization and Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/K2EENXT5}},
note = {Machine review of arXiv:2507.22322}
}
read the original abstract
Sound Event Localization and Detection (SELD) is crucial in spatial audio processing, enabling systems to detect sound events and estimate their 3D directions. Existing SELD methods use single- or dual-branch architectures: single-branch models share SED and DoA representations, causing optimization conflicts, while dual-branch models separate tasks but limit information exchange. To address this, we propose a two-step learning framework. First, we introduce a tracwise reordering format to maintain temporal consistency, preventing event reassignments across tracks. Next, we train SED and DoA networks to prevent interference and ensure task-specific feature learning. Finally, we effectively fuse DoA and SED features to enhance SELD performance with better spatial and event representation. Experiments on the 2023 DCASE challenge Task 3 dataset validate our framework, showing its ability to overcome single- and dual-branch limitations and improve event classification and localization.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction The Sound Event Localization and Detection (SELD) system detects sound events in three-dimensional space, tracks their temporal activity, and estimates their direction of arrival. These systems are widely used in real-world applications, including crime detection, smart home monitoring, and human-robot in- teraction, enhancing situational awa...
work page 2019
-
[2]
A Two-Step Learning Framework for Enhancing Sound Event Localization and Detection
Proposed Method Our proposed method follows a two-step learning framework to optimize SELD performance. First, we independently train DoA and SED networks to prevent interference. We adopt IPDnet[16] and ATST-SED[17] for DoA and SED, extending their architectures to fit the polyphonic SELD scenario. We pro- pose a trackwise output format to prevent event ...
work page Pith review arXiv 2025
-
[3]
Experiments 3.1. Implementation Details For training, we utilized the STARSS23[25] dev-set-train and a synthetic dataset, while evaluation was conducted on the dev- set-test. The dataset provides two recording formats: first-order ambisonics (FOA) and microphone array (MIC). The proposed model was trained using different input configurations for DoA and S...
work page 2023
-
[4]
Conclusion We propose a two-step learning framework for SELD, integrat- ing structured fusion to leverage both SED and DoA features effectively. The trackwise reordering format and beamformed spatial features enhance temporal consistency and localization accuracy. Our model achieves the lowest SELD score on the 2023 DCASE Task 3 dataset, showing its effec...
work page 2023
-
[5]
S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using con- volutional recurrent neural networks,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 1, pp. 34–48, 2018
work page 2018
-
[6]
The ustc-iflytek system for sound event localization and detection of dcase2020 challenge,
Q. Wang, H. Wu, Z. Jing, F. Ma, Y . Fang, Y . Wang, T. Chen, J. Pan, J. Du, and C.-H. Lee, “The ustc-iflytek system for sound event localization and detection of dcase2020 challenge,” IEEE AASP Chall. Detect. Classif. Acoust. Scenes Events, 2020
work page 2020
-
[7]
Overview and evaluation of sound event localization and detec- tion in dcase 2019,
A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, “Overview and evaluation of sound event localization and detec- tion in dcase 2019,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 684–698, 2021
work page 2019
-
[8]
Seld-tcn: Sound event localization & detection via temporal con- volutional networks,
K. Guirguis, C. Schorn, A. Guntoro, S. Abdulatif, and B. Yang, “Seld-tcn: Sound event localization & detection via temporal con- volutional networks,” in 2020 28th European Signal Processing Conference (EUSIPCO), 2021, pp. 16–20
work page 2020
Show all 32 references
-
[9]
An improved event-independent network for polyphonic sound event localization and detection,
Y . Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” inICASSP 2021-2021 IEEE In- ternational Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2021, ...
2021
-
[10]
A track-wise ensemble event independent network for polyphonic sound event localization and detection,
J. Hu, Y . Cao, M. Wu, Q. Kong, F. Yang, M. D. Plumbley, and J. Yang, “A track-wise ensemble event independent network for polyphonic sound event localization and detection,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)....
2022
-
[11]
Sound event localization and detection for real spatial sound scenes: Event-independent network and data augmentation chains,
——, “Sound event localization and detection for real spatial sound scenes: Event-independent network and data augmentation chains,” arXiv preprint arXiv:2209.01802, 2022
2022 arXiv
-
[12]
Mff-einv2: Multi-scale feature fu- sion across spectral-spatial-temporal domains for sound event lo- calization and detection,
D. Mu, Z. Zhang, and H. Yue, “Mff-einv2: Multi-scale feature fu- sion across spectral-spatial-temporal domains for sound event lo- calization and detection,”arXiv preprint arXiv:2406.08771, 2024
2024 arXiv
-
[13]
Seld- mamba: Selective state-space model for sound event localization and detection with source distance estimation,
D. Mu, Z. Zhang, H. Yue, Z. Wang, J. Tang, and J. Yin, “Seld- mamba: Selective state-space model for sound event localization and detection with source distance estimation,” arXiv preprint arXiv:2408.05057, 2024
2024 arXiv
-
[14]
Accdoa: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,
K. Shimada, Y . Koyama, N. Takahashi, S. Takahashi, and Y . Mit- sufuji, “Accdoa: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Proce...
2021
-
[15]
Multi-accdoa: Localizing and detecting over- lapping sounds from the same class with auxiliary duplicating per- mutation invariant training,
K. Shimada, Y . Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y . Mitsufuji, “Multi-accdoa: Localizing and detecting over- lapping sounds from the same class with auxiliary duplicating per- mutation invariant training,” in ICASSP 2022 - 2022 IEEE Inter- national Conferenc...
2022
-
[16]
An experimental study on sound event localization and detection under realistic testing conditions,
S. Niu, J. Du, Q. Wang, L. Chai, H. Wu, Z. Nian, L. Sun, Y . Fang, J. Pan, and C.-H. Lee, “An experimental study on sound event localization and detection under realistic testing conditions,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Pro...
2023
-
[17]
Cst-former: Transformer with channel- spectro-temporal attention for sound event localization and detec- tion,
Y . Shul and J.-W. Choi, “Cst-former: Transformer with channel- spectro-temporal attention for sound event localization and detec- tion,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 8686–8690
2024
-
[18]
The cocktail party problem,
S. Haykin and Z. Chen, “The cocktail party problem,” Neural computation, vol. 17, no. 9, pp. 1875–1902, 2005
1902
-
[19]
Doa and event guidance system for sound event localiza- tion and detection with source distance estimation,
H. Yu, “Doa and event guidance system for sound event localiza- tion and detection with source distance estimation,” DCASE2024 Challenge, Tech. Rep, Tech. Rep., 2024
2024
-
[20]
Ipdnet: A universal direct-path ipd estimation network for sound source localization,
Y . Wang, B. Yang, and X. Li, “Ipdnet: A universal direct-path ipd estimation network for sound source localization,” arXiv preprint arXiv:2405.07021, 2024
2024 arXiv
-
[21]
Fine-tune the pretrained atst model for sound event detection,
N. Shao, X. Li, and X. Li, “Fine-tune the pretrained atst model for sound event detection,” in ICASSP 2024-2024 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 911–915
2024
-
[22]
The cone of silence: Speech separation by local- ization,
T. Jenrungrot, V . Jayaram, S. Seitz, and I. Kemelmacher- Shlizerman, “The cone of silence: Speech separation by local- ization,” Advances in Neural Information Processing Systems , vol. 33, pp. 20 925–20 938, 2020
2020
-
[23]
Rezero: Region-customizable sound extrac- tion,
R. Gu and Y . Luo, “Rezero: Region-customizable sound extrac- tion,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
2024
-
[24]
Lavt: Language-aware vision transformer for referring image segmentation,
Z. Yang, J. Wang, Y . Tang, K. Chen, H. Zhao, and P. H. Torr, “Lavt: Language-aware vision transformer for referring image segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 155– 18 165
2022
-
[25]
Separate and re- construct: Asymmetric encoder-decoder for speech separation,
U.-H. Shin, S. Lee, T. Kim, and H.-M. Park, “Separate and re- construct: Asymmetric encoder-decoder for speech separation,” arXiv preprint arXiv:2406.05983, 2024
2024
-
[26]
[dcase2022 task 3] synthetic seld mixtures for baseline training,
A. Politis, “[dcase2022 task 3] synthetic seld mixtures for baseline training,” Synthetic SELD mixtures for baseline training, 2022
2022
-
[27]
Conformer-based sound event detection with semi- supervised learning and data augmentation,
K. Miyazaki, T. Komatsu, T. Hayashi, S. Watanabe, T. Toda, and K. Takeda, “Conformer-based sound event detection with semi- supervised learning and data augmentation,” dim, vol. 1, no. 4, 2020
2020
-
[28]
Ast: Audio spectrogram transformer,
Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” arXiv preprint arXiv:2104.01778, 2021
2021 arXiv
-
[29]
Starss23: Sony-tau realistic spatial soundscapes 2023,
A. Politis, K. Shimada, P. Sudarsanam, A. Hakala, S. Takahashi, D. Krause, N. Takahashi, S. Adavanne, Y . Koyama, K. Uchida et al. , “Starss23: Sony-tau realistic spatial soundscapes 2023,” 2023
2023
-
[30]
Permutation invari- ant training of deep models for speaker-independent multi-talker speech separation,
D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen, “Permutation invari- ant training of deep models for speaker-independent multi-talker speech separation,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 241–245
2017
-
[31]
Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,
K. Shimada, A. Politis, P. Sudarsanam, D. A. Krause, K. Uchida, S. Adavanne, A. Hakala, Y . Koyama, N. Takahashi, S. Takahashi et al., “Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,”Ad- vances in Neural ...
2024
-
[32]
Divided spectro-temporal at- tention for sound event localization and detection in real scenes for dcase2023 challenge,
Y . Shul, B.-Y . Ko, and J.-W. Choi, “Divided spectro-temporal at- tention for sound event localization and detection in real scenes for dcase2023 challenge,” arXiv preprint arXiv:2306.02591 , 2023
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.