REVIEW 3 major objections 5 minor 2 cited by
Description and Discussion on DCASE 2025 Challenge Task 4: Spatial Semantic Segmentation of Sound Scenes
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Joint detection and separation of spatial sound events is made measurable by a new challenge dataset, a label-aware metric, and baseline systems, with the best baseline scoring 6.60 dB on the held-out evaluation set.
desk verdict Genuinely new benchmark data and a usable baseline, undermined by an underspecified metric for duplicate classes and a thin two-row proportionality claim; worth one round of referee attention. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs through three connected pieces. The generative mixture model of Eq. (1) writes each microphone channel as the sum of RIR-convolved dry sources plus noise, which is what the simulator renders; the training target is the relaxed, direct-path version of each source, defined as the source convolved with the direct-path impulse response to a reference microphone, so systems learn spatial cues without solving propagation-delay compensation. The evaluation machinery is CA-SDRi (Eqs. (2)-(5)), a label-aligned signal-to-distortion ratio improvement: for each label in the union of true and predicted sets it awards SDRi when the label is correct, and zero for false positives and false negatives, then averages over the clip. The baseline systems are two-stage: a masked-modeling audio tagging model (M2D) classifies events, and either a single-source ResUNet or its multi-source variant ResUNetK performs separation, with ResUNetK querying several sources concurrently.
What would settle it
Record genuine multi-channel mixtures of the same 18 event classes in real rooms with ground-truth direct-path references, run the released ResUNetK baseline on them, and compare its CA-SDRi with the 6.60 dB evaluation-set score; a large shortfall would show the synthetic benchmark does not transfer to real acoustic environments.
Extended reading notes
Core claim
On its own terms, this paper establishes a reproducible evaluation protocol for Spatial Semantic Segmentation of Sound Scenes (S5): given a multi-channel mixture of at most three co-occurring events plus interference and ambient noise, a system must return one isolated source signal and one class label per event. The paper defines the training target as the event's signal convolved with the direct-path impulse response toward a reference microphone, which keeps spatial cues while deferring distance and reverberation compensation. It then presents the DCASE2025 Task4 Dataset of 18 classes, built from newly recorded anechoic one-shot events, measured RIRs in three rooms, curated noise and interference, and simulator-rendered 10-second mixtures, and shows that the ResUNetK baseline reaches 6.60 dB CA-SDRi on the evaluation set (ResUNet: 5.72 dB, same 51.48% classification accuracy). The associated metric, CA-SDRi, aligns estimated and reference sources by label and assigns zero to mislabels and missed events, and the paper reports that CA-SDRi varies with classification accuracy, supporting its use as the joint ranking score.
Load-bearing premise
All of the benchmark's scores are produced on mixtures synthesized by a simulator from anechoic one-shot recordings, measured impulse responses, and curated noise and interference, so the entire enterprise assumes those synthetic mixtures behave enough like real spatial sound scenes that evaluation scores predict real-world performance.
Editorial extensions
If this is right
- CA-SDRi rewards only correctly labeled and well-separated events, so a system's ranking score reflects detection, classification, and separation quality together rather than separation alone.
- ResUNetK outperforms ResUNet on both evaluation and development test sets despite sharing the same classifier, showing that multi-source querying in the separation stage is the source of the gain.
- The evaluation set is constructed entirely from newly recorded material, so reported scores are not inflated by overlap with public training data.
- The dataset provides a controlled testbed with 18 event classes, up to three simultaneous target events, one or two interference sounds, and per-event SNRs of 5-20 dB against background noise, useful for isolating the effect of each factor.
- Because target signals are direct-path convolutions to a reference microphone, systems are not required to solve full dereverberation or distance estimation, keeping this year's benchmark focused on detection and separation.
Reading between the lines
- A natural extension beyond this year's task is to include direction-of-arrival and distance metadata, moving from the direct-path target back toward true dry source objects with full 6DoF spatial information; the paper names this as the ultimate goal but leaves it out of the current task.
- Because false positives and false negatives contribute zero to CA-SDRi rather than negative penalties, a system that outputs nothing receives the same score as one that guesses wrong; future ranking schemes could reward partially correct detections or penalize confident wrong labels.
- The gap between the development test subset (CA-SDRi about 11 dB) and the evaluation set (about 6.6 dB) suggests the newly recorded evaluation materials are harder; probing which factors, such as unseen rooms, microphone arrays, or event classes, drive that gap would guide dataset design.
- A testable extension is to increase the maximum number of simultaneous events beyond three and to add moving sources; the current benchmark is a static, bounded cocktail-party regime, and it is unknown whether the same architectures scale.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes the DCASE 2025 Challenge Task 4, Spatial Semantic Segmentation of Sound Scenes (S5). It defines the task as detecting and separating up to three simultaneous sound events from multichannel spatial mixtures, with each output being an isolated source signal and a class label. The paper introduces the newly recorded DCASE2025 Task 4 dataset, comprising anechoic one-shot event recordings, measured room impulse responses, environmental noise, and interference sounds, and explains the synthesis of training and evaluation mixtures. It also defines the class-aware signal-to-distortion ratio improvement (CA-SDRi) metric and reports baseline results for two systems, ResUNet and ResUNetK. The paper states that a full version with challenge results will appear later.
Significance. Conditional on the task specification being tightened, the paper provides a useful public benchmark for joint sound event detection and source separation with spatial input. Its concrete strengths are the release of newly recorded development and evaluation data on Zenodo and baseline code on GitHub, which make the descriptive claims verifiable. The experimental contribution is modest: two baseline systems are compared using single runs, and the claimed proportionality between CA-SDRi and classification accuracy is not supported by the reported table. The main formal risk is that the metric is defined over label sets and is under-specified for clips with repeated classes, which the generation recipe does not explicitly forbid.
major comments (3)
- [Sec. 2, Eq. (2)-(3), Sec. 3.2] The metric is defined over the set C of source labels, and Eq. (2) sums one term per element of C ∪ Ĉ. Section 3.2 states only that a clip contains one to three sound events and never states that target classes are sampled without replacement. If two events of the same class occur, C collapses them into one label, so Eq. (2) cannot count both instances, Eq. (3) cannot align estimated and reference instances, and the denominator |C ∪ Ĉ| undercounts false positives and false negatives. Because CA-SDRi is the ranking score in Table 1, this makes the headline baseline comparison formally undefined for a permissible input. Please either state explicitly that classes are unique within a clip and verify this in the generation recipe, or reformulate C as a multiset and adapt the metric accordingly.
- [Sec. 5, Table 1] The text claims that "the CA-SDRi is shown to vary proportionally with classification accuracy," but Table 1 reports identical classification accuracy for ResUNetK and ResUNet on both the evaluation and development sets (51.48 and 59.80, respectively) while CA-SDRi differs (6.60 vs. 5.72 and 11.09 vs. 11.03). The reported data therefore do not support a proportionality claim; at most they show that CA-SDRi can differ when accuracy is tied. This sentence should be removed or replaced with a statement about the two specific points.
- [Sec. 5, Table 1] The baseline ranking is based on single runs without error bars, confidence intervals, or a statement of the number of evaluation clips. Since the CA-SDRi gap is about 0.9 dB on the evaluation set and 0.06 dB on the development set, the reader cannot tell whether the ResUNetK advantage is meaningful. Please report run-to-run variance, or explicitly state that each score comes from a single run and treat the comparison as anecdotal.
minor comments (5)
- [Table 1] The table header is malformed: "Evaluation setDevelopment set" runs together; please split it into separate column headers.
- [Abstract and Sec. 1] The abstract mentions "6 Degrees of Freedom (6DoF) information" and "dry sound object signals," but the task in this year's challenge does not use 6DoF; please define or remove these terms to avoid overclaiming.
- [Sec. 3.3] The section says further details will appear in a later version; since the evaluation set is already released, including at least basic statistics such as the number of clips and event counts would make the paper more self-contained.
- [Sec. 4] In Eq. (2) and Eq. (3), P_{c_k} is used for a metric component while c_k appears both as a label and as an index; consider writing P(c_k) for readability.
- [Introduction] The acronym S5 is used without expansion beyond the abstract; please expand it at first use in the introduction.
Circularity Check
No central circularity; baseline results are measured. One minor definitional over-claim in the metric's self-validation.
-
self definitional
[Section 5, Experimental Results and Discussion (Table 1 discussion)]
"In addition, the CA-SDRi is shown to vary proportionally with classification accuracy, further validating its efficacy in simultaneously evaluating both source separation and class label prediction in the S5 task."
In Eq. (2), CA-SDRi is the average of Pck over C∪Ĉ, and Eq. (3) sets Pck = SDRi only when ck ∈ C∩Ĉ and Pck = 0 otherwise. Thus CA-SDRi = (1/|C∪Ĉ|) Σ_{ck∈C∩Ĉ} SDRi(x̂k, xk, y). When per-event SDRi values are similar, this equals SDRi × |C∩Ĉ|/|C∪Ĉ|, i.e., a label-overlap accuracy term scaled by a constant. The reported proportionality with classification accuracy is therefore a built-in property of the metric definition, not an independent empirical validation of the metric's efficacy.
full rationale
The paper is a challenge description with baseline measurements, not a derivation. The DCASE2025 Task 4 dataset is newly recorded for the evaluation split (Sec. 3.3), and Table 1 reports measured CA-SDRi and accuracy values for ResUNet and ResUNetK, so these numbers are experimental outputs rather than fitted inputs. Self-citations to [5] define the task and baseline architecture, but the paper restates the equations (Secs. 2 and 4) and does not rely on an unstated self-cited theorem. The PFN=PFP=0 choice in Eq. (3) is an explicit design decision. One minor self-definitional over-claim occurs in Sec. 5: the statement that CA-SDRi 'is shown to vary proportionally with classification accuracy' is partly a consequence of Eq. (2)-(3), because with zero penalties the metric is the average SDRi over correctly labeled events; if SDRi is roughly constant, CA-SDRi is proportional to the label-overlap ratio |C∩Ĉ|/|C∪Ĉ|. This makes the correlation with accuracy an expected property of the metric, not an independent validation, but it does not affect the central benchmark results. The skeptic's duplicate-class concern (C defined as a set while Sec. 3.2 allows one to three events without stating class uniqueness) is a real metric under-specification risk, but it is a correctness issue rather than a circular reduction. Overall, no load-bearing prediction reduces to its inputs; score 2 for the minor definitional validation and self-referential baselines.
Assumptions & free parameters
free parameters (4)
- Maximum simultaneous events Kmax=3 =
3
- Target-event SNR range =
5 to 20 dB
- Interference SNR range =
0 to 15 dB
- Metric penalty values PFN/PFP =
0
assumptions (3)
- domain assumption The observed multichannel signal is a linear convolution mixture of dry sources with room impulse responses plus additive noise (Eq. 1).
- ad hoc to paper Target signals are defined as the source convolved with the direct-path impulse response to a reference microphone, rather than the true dry source.
- domain assumption All RIRs in a mixture must come from the same microphone position in the same room, and mixtures contain at most three events plus one or two interferers.
Cite this review
Pith. "Pith review of Description and Discussion on DCASE 2025 Challenge Task 4: Spatial Semantic Segmentation of Sound Scenes." pith.science (2026). https://pith.science/paper/DMI267JL
@misc{pith2026250610676,
author = {Pith},
title = {Pith review of: Description and Discussion on DCASE 2025 Challenge Task 4: Spatial Semantic Segmentation of Sound Scenes},
year = {2026},
howpublished = {\url{https://pith.science/paper/DMI267JL}},
note = {Machine review of arXiv:2506.10676}
}
read the original abstract
Spatial Semantic Segmentation of Sound Scenes (S5) aims to enhance technologies for sound event detection and separation from multi-channel input signals that mix multiple sound events with spatial information. This is a fundamental basis of immersive communication. The ultimate goal is to separate sound event signals with 6 Degrees of Freedom (6DoF) information into dry sound object signals and metadata about the object type (sound event class) and representing spatial information, including direction. However, because several existing challenge tasks already provide some of the subset functions, this task for this year focuses on detecting and separating sound events from multi-channel spatial input signals. This paper outlines the S5 task setting of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2025 Challenge Task 4 and the DCASE2025 Task 4 Dataset, newly recorded and curated for this task. We also report experimental results for an S5 system trained and evaluated on this dataset. The full version of this paper will be published after the challenge results are made public.
Figures
Forward citations
Cited by 2 Pith papers
-
On Temporal Guidance and Iterative Refinement in Audio Source Separation
A DCASE 2025 Task 4 system that guides audio source separation with frame-level event detection and iterative refinement, reaching second place.
-
Performance improvement of spatial semantic segmentation with enriched audio features and agent-based error correction for DCASE 2025 Challenge Task 4
A combination of extra audio features, curated training data, and agent-based label correction raises CA-SDRi from 11.088 dB to 12.726 dB on DCASE 2025 Task 4.
Reference graph
Works this paper leans on
-
[5]
EXPERIMENTAL RESULTS AND DISCUSSION This section presents the performance of the baseline systems, Re- sUNet and ResUNetK, proposed in [5]. Both are two-stage systems System Evaluation set Development setRanking Score (CA-SDRi)↑ Acc.↑ CA-SDRi↑ Acc.↑Nguyen_NTT_task4_1 (ResUNetK)6.60 51 .48 11 .09 59 .80Nguyen_NTT_task4_2 (ResUNet)5.72 51 .48 11 .03 59 .80 ...
-
[1]
INTRODUCTION This paper summarizes a newly introduced task to the Detection and Classification of Acoustic Scenes and Events (DCASE) 2025 challenge, named Task 4: Spatial Semantic Segmentation of Sound Scenes (S5) [1], and discusses the results of the challenge. This task requires systems to detect and extract sound events from multi-channel spatial input...
work page 2025
-
[2]
This section introduces the notation and task settings for the S5 task
TASK SETTING OF S5 [5] The S5 task, originally proposed in our prior work [5], aims to de- tect and separate the sounds of each sound event from signals ob- served by a multi-channel microphone at various locations in a real environment. This section introduces the notation and task settings for the S5 task. Let Y = [y(1), . . . ,y(M )]⊤ ∈ RM ×T be the mu...
-
[3]
DCASE2025 TASK4 DA TASET 3.1. General overview For the S5 task, we designed and recorded a new dataset named DCASE2025 Task4 Dataset [6, 7]. Given the task definition in Sec. 2, a dataset suitable for S5 requires the following resources: arXiv:2506.10676v1 [cs.SD] 12 Jun 2025 Detection and Classification of Acoustic Scenes and Events 2025 29 + 30-31 Octob...
arXiv 2025
-
[4]
EV ALUA TION METHOD, METRIC [5] As evaluation metrics for the S5 task, we used class-aware signal- to-distortion ratio improvement (CA-SDRi) proposed for the S5 task [5]. The key concept of CA-SDRi is that the estimated and reference sources are aligned by their labels, with the waveform metric being calculated when the label is correctly predicted, i.e.,...
-
[6]
ACKNOWLEDGMENT This work was partially supported by JST Strategic International Collaborative Research Program (SICORP), Grant Number JP- MJSC2306, Japan. This work was partially supported by the Agence Nationale de la Recherche (Project Confluence, grant number ANR-23-EDIA- 0003)
- [7]
-
[8]
Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,
N. Turpault, R. Serizel, A. Parag Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in Workshop on Detection and Classification of Acoustic Scenes and Events, New York City, United States, October 2019. [Online]. Available: https://hal.inria.fr/hal-02160855
work page 2019
Show all 24 references
-
[9]
[Online]
(2021) DCASE2021 challenge task 4: Sound event detection and separation in domestic environments. [Online]. Avail- able: https://dcase.community/challenge2021/task-sound- event-detection-and-separation-in-domestic-environments
2021
-
[10]
[Online]
(2024) DCASE2024 challenge task 3: Audio and audiovisual sound event localization and detection with source distance estimation. [Online]. Available: https://dcase.community/challenge2024/task-audio-and- audiovisual-sound-event-localization-and-detection-with- source-distance-...
2024
-
[11]
Baseline systems and evaluation metrics for spatial semantic segmentation of sound scenes,
B. T. Nguyen, M. Yasuda, D. Takeuchi, D. Niizumi, Y . Ohishi, and N. Harada, “Baseline systems and evaluation metrics for spatial semantic segmentation of sound scenes,” in 2025 33rd European Signal Processing Conference (EUSIPCO) , 2025. [Online]. Available: https://arxiv.org...
2025 arXiv
-
[12]
[Online]
(2025) DCASE2025 Task 4 S5 Development set. [Online]. Available: https://zenodo.org/records/15117227
2025
-
[13]
[Online]
(2025) DCASE2025 Task 4 S5 Evaluation set. [Online]. Available: https://zenodo.org/records/15553984
2025
-
[14]
FSD50K: an open dataset of human-labeled sound events,
E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: an open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2021
2021
-
[15]
Semantic hearing: Programming acoustic scenes with binau- ral hearables,
B. Veluri, M. Itani, J. Chan, T. Yoshioka, and S. Gollakota, “Semantic hearing: Programming acoustic scenes with binau- ral hearables,” in Proceedings of the 36th Annual ACM Sym- posium on User Interface Software and Technology, 2023, pp. 1–15
2023
-
[16]
Echo-aware adaptation of sound event localization and detection in unknown environ- ments,
M. Yasuda, Y . Ohishi, and S. Saito, “Echo-aware adaptation of sound event localization and detection in unknown environ- ments,” in IEEE Intl. Conf. on Acoust., Speech & Sig. Proc. (ICASSP), 2022, pp. 226–230
2022
-
[17]
[Online]
(2025) DCASE2025 Task 4 Baseline. [Online]. Available: https://github.com/nttcslab/dcase2025_task4_baseline
2025
-
[18]
Spatial scaper: a library to simulate and augment soundscapes for sound event localization and detection in re- alistic rooms,
I. R. Roman, C. Ick, S. Ding, A. S. Roman, B. McFee, and J. P. Bello, “Spatial scaper: a library to simulate and augment soundscapes for sound event localization and detection in re- alistic rooms,” in IEEE Intl. Conf. on Acoust., Speech & Sig. Proc. (ICASSP), 2024, pp. 1221–1225
2024
-
[19]
Ears: An anechoic full- band speech dataset benchmarked for speech enhancement and dereverberation,
J. Richter, Y .-C. Wu, S. Krenn, S. Welker, B. Lay, S. Watan- abe, A. Richard, and T. Gerkmann, “Ears: An anechoic full- band speech dataset benchmarked for speech enhancement and dereverberation,” in Interspeech 2024, 2024, pp. 4873– 4877
2024
-
[20]
Percep- tual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,
A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Percep- tual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat....
2001
-
[21]
A short-time objective intelligibility measure for time-frequency weighted noisy speech,
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short-time objective intelligibility measure for time-frequency weighted noisy speech,” in 2010 IEEE International Confer- ence on Acoustics, Speech and Signal Processing , 2010, pp. 4214–4217
2010
-
[22]
Masked modeling duo: Towards a universal audio pre-training framework,
D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Towards a universal audio pre-training framework,” IEEE/ACM Trans. on Audio, Speech, and Lang. Process., 2024
2024
-
[23]
Universal source separation with weakly labelled data,
Q. Kong, K. Chen, H. Liu, X. Du, T. Berg-Kirkpatrick, S. Dubnov, and M. D. Plumbley, “Universal source separation with weakly labelled data,” arXiv preprint arXiv:2305.07447, 2023
2023 arXiv
-
[24]
[On- line]
(2025) Github: nttcslab/dcase2025_task4_baseline. [On- line]. Available: https://github.com/nttcslab/dcase2025_ task4_baseline
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.