REVIEW 4 major objections 5 minor 29 references
Ensemble Confidence Calibration for Sound Event Detection in Open-environment
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Weighting an SED ensemble by calibrated occurrence-confidences lifts out-of-domain F1 by roughly half.
desk verdict A promising but under-specified ensemble weighting scheme: large gains on one OOD split, but c_m(x) is never defined and calibration isn't measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the EOW-Softmax classifier on the SOD branch: a $(P+1)$-way softmax where $P=3$ states (no target event, monophonic event, polyphonic event) plus an extra open-world uncertainty class. Training uses a maximum-likelihood term on in-domain SOD labels and an auxiliary term on pseudo-out-of-domain features sampled from normal distributions fitted by stochastic gradient Langevin dynamics (SGLD). The confidence $c_m(x)$ is the calibrated probability from this head, and it does the work of a per-model reliability weight in the ensemble fusion formula, Equation (4).
What would settle it
Measure the Spearman correlation between the SOD confidence $c_m(x)$ and the per-frame accuracy of the SED branch on the out-of-domain test set; if it is close to zero or negative, the ensemble gain cannot be attributed to confidence-guided weighting. A second falsifier is to shuffle the $c_m(x)$ values across ensemble members and see whether the F1 scores remain essentially unchanged.
Extended reading notes
Core claim
The central claim is that confidence scores from a sound occurrence and overlap detection (SOD) branch, trained with EOW-Softmax, reliably indicate how much each ensemble member should be trusted for the sound event detection (SED) task. The paper shows this in Equation (4), which weights each model's SED output by the SOD confidence $c_m(x)$, and in Table I, where the calibrated ensemble beats both the single model and an equal-weight average ensemble on all four F1 metrics. The authors attribute the gain to two effects: EOW-Softmax creates an explicit open-world uncertainty class, so uncertain frames yield low confidence weights; and the calibrated weighting prevents a few overconfident models from dominating the fusion. A secondary claim is that EOW-Softmax also regularizes training, since the single-model result (P2) already improves over the baseline (P1).
Load-bearing premise
The whole method rests on the premise that the SOD branch's confidence scores actually track how correct the SED branch's predictions are on a given input; the paper never directly checks this correlation.
Editorial extensions
If this is right
- In an open-environment test set with training and testing scenes drawn from different datasets, the calibrated ensemble outperforms both the best single model and the unweighted average ensemble on event-based and segment-based F1.
- The largest gains appear on short, rare, or acoustically confusable events such as bird singing, children, and rustling, where calibrated confidence prevents overconfident members from dragging the ensemble down.
- Because the SOD branch explicitly classifies the polyphonic state, the weighting scheme can improve robustness without sacrificing the detection of overlapping events.
- The approach requires no changes to the SED architecture beyond adding the SOD head, making it directly applicable to standard convolutional recurrent sound-event baselines.
Reading between the lines
- The paper does not directly measure the correlation between SOD confidence and SED per-frame correctness; an ablation that shuffles or randomizes the $c_m(x)$ weights would tell whether the gain comes from confidence-guided fusion or merely from EOW-Softmax's regularizing effect on each member model.
- If the calibration transfer is real, the same SOD confidence could be reused for test-time adaptation, for example up-weighting frames or models on known target scenes, or rejecting frames whose open-world probability is high.
- A stronger test would compare SOD-confidence weighting against weighting by calibrated SED logits; nearly identical gains there would show that the SOD head is a convenient but not unique source of confidence.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an ensemble method for sound event detection (SED) in open environments. A dual-branch model is trained with the EOW-Softmax objective on a sound occurrence and overlap detection (SOD) auxiliary task; at test time, multiple models' SED branch outputs are combined by weighting with a confidence value c_m(x) from each model's SOD branch (Eq. 4). Experiments on a TUT 2016/2017 out-of-domain split (training on 'home' and 'residential area', testing on 'city center' and 'office') show that the proposed combination (P4) improves all four F-scores relative to the baseline (P1), the unweighted ensemble (P3), and the single EOW-Softmax model (P2).
Significance. If the improvement is real and is actually caused by confidence calibration, this is a practically useful contribution: a simple weighting scheme lifts event-based and segment-based F-scores substantially in an out-of-domain setting, and the paper is among the first to combine EOW-Softmax-style uncertainty modeling with ensemble fusion for SED. The study is clearly framed, compares four configurations, and uses standard SED metrics. However, the central mechanism is not yet demonstrated: the quantity c_m(x) used as the ensemble weight is never defined, no calibration metric (ECE, reliability diagram) is reported despite the overconfidence-reduction claim, and the experimental evidence rests on a single domain split with averaged point estimates. The contribution is therefore plausible but under-supported in its current form.
major comments (4)
- [Section II-C, Eq. (4)] The confidence weight c_m(x) is never defined. The text states only that it is the 'confidence calibration learned by the SOD branch' of model m, but it does not specify whether c_m(x) is the maximum softmax probability over the three SOD states, the complement of the open-world uncertainty dimension, or some other scalar, nor whether it is computed per frame, per clip, or after temporal smoothing. This is not a minor detail: Eq. (4) normalizes the weights, so any monotone transformation of c_m(x) changes the ensemble output. Without an explicit formula, the method is not reproducible and the Table I gains for P4 cannot be attributed to confidence-based weighting.
- [Section IV and Abstract] The paper repeatedly claims that the method 'reduces overconfidence' and performs 'confidence calibration', but no calibration metric is reported. I ask the authors to include expected calibration error (ECE), reliability diagrams, or at least a comparison of confidence against accuracy for P1 versus P4 on both in-domain and out-of-domain test sets. Without such evidence, the central interpretation of the ensemble gain as a calibration effect remains speculative; the improvement could come from variance reduction or from another form of model selection.
- [Section III-B and Table I] The experimental protocol described in Section III-B is ambiguous. The text says 'The final model result is the average of the best checkpoints from these five runs' and then separately says 'For model ensembling, each candidate model is given an equal interpolation weight of 0.2'. If the baseline P1 already averages five checkpoints, then it is itself an ensemble, which conflicts with the interpretation of P3 as the 'average linear ensemble'. Please clarify which rows in Table I correspond to single models and which correspond to ensembles, and describe exactly how the five trained models are used for each of P1 through P4.
- [Table I and Section III-B] All results are reported as point estimates from a single out-of-domain split, with five-seed averages but no variance, error bars, or significance tests. Because the central claims depend on the margins between P4 and the other configurations (e.g., Smi-F1 57.92 vs. 43.34 for P2), it is important to know the per-seed spread and whether the differences are consistent. I request reporting the standard deviation or per-seed results, and ideally evaluation on more than one domain split (e.g., leave-one-scene-out) to show that the gains are not an artifact of the particular 'city center + office' test set.
minor comments (5)
- [Section II-B, Eq. (2)] The notation 'Ψϕ(x) ∥i∥' is unusual and is not defined; please use conventional subscript notation such as Ψϕ(x)_i.
- [Abstract and Introduction] The claim 'we are the first to use ensemble methods in SED' is contradicted by the paper's own references [14]–[18], which already propose ensemble approaches for SED. Please qualify the novelty statement to refer to open-environment/out-of-domain robustness with calibration-based weighting.
- [Section III-B] The hyperparameters of EOW-Softmax (λ in Eq. (3), SGLD settings, τ in Eq. (1)) are said to 'follow the same configuration as used in [25]'. This is insufficient for a self-contained reproducibility; please provide the values or a precise pointer to the configuration in [25].
- [General] Several typos appear: 'Modle 1/Modle 2' in Figure 2, repeated 'REFERENCES' heading in the bibliography, and inconsistent capitalization of 'Eow-Softmax' versus 'EOW-Softmax'. These should be corrected.
- [Figure 4] Figure 4 reports per-event Smi-F1 scores without error bars or number of runs. Adding variance information would help interpret the improvements on rare events such as 'bs' and 'sq'.
Circularity Check
No circularity: the ensemble weighting is a design choice, and the F-scores are measured on independent held-out scenes; the undefined c_m(x) is a reproducibility gap, not a circular step.
full rationale
The derivation chain is not circular. Equation (4) is a design choice: the final SED prediction is a confidence-weighted arithmetic combination of independently trained SED branch outputs, and the reported F-scores are computed on an out-of-domain acoustic-scene split that is held out from training, so the predicted quantity is not defined in terms of a fitted parameter or in terms of the test labels. EOW-Softmax is imported from external prior work (Ref. [25]) and applied to the SOD branch; although the paper never gives the explicit functional form of the SOD confidence weight c_m(x) in Eq. (4), that omission is a reproducibility and validation gap rather than a circular reduction, since no equation defines c_m through the SED prediction it is used to weight. The absence of calibration metrics (e.g., expected calibration error or reliability diagrams) weakens the causal claim that the P4 gain is solely due to calibrated confidence, and the modest self-citations to the co-author's earlier work (Refs. [13] and [21]) are related-work background, not load-bearing premises. Because no step introduces the target result as an input, the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Ensemble interpolation weight =
0.2 per model (equal)
- EOW-Softmax hyperparameters =
same as [25], not specified here
- LOPEN weight tau =
not stated
- Soft mixup Beta parameters =
alpha=0.2, beta=0.2
assumptions (3)
- domain assumption SOD branch confidence c_m(x) is a valid reliability weight for SED predictions.
- domain assumption EOW-Softmax's extra open-world dimension captures out-of-domain uncertainty.
- domain assumption The TUT scene split (home and residential area vs. city center and office) is representative of open-environment domain shift.
Cite this review
Pith. "Pith review of Ensemble Confidence Calibration for Sound Event Detection in Open-environment." pith.science (2026). https://pith.science/paper/5ICQPM5E
@misc{pith2026250709606,
author = {Pith},
title = {Pith review of: Ensemble Confidence Calibration for Sound Event Detection in Open-environment},
year = {2026},
howpublished = {\url{https://pith.science/paper/5ICQPM5E}},
note = {Machine review of arXiv:2507.09606}
}
read the original abstract
Sound event detection (SED) has made strong progress in controlled environments with clear event categories. However, real-world applications often take place in open environments. In such cases, current methods often produce predictions with too much confidence and lack proper ways to measure uncertainty. This limits their ability to adapt and perform well in new situations. To solve this problem, we are the first to use ensemble methods in SED to improve robustness against out-of-domain (OOD) inputs. We propose a confidence calibration method called Energy-based Open-World Softmax (EOW-Softmax), which helps the system better handle uncertainty in unknown scenes. We further apply EOW-Softmax to sound occurrence and overlap detection (SOD) by adjusting the prediction. In this way, the model becomes more adaptable while keeping its ability to detect overlapping events. Experiments show that our method improves performance in open environments. It reduces overconfidence and increases the ability to handle OOD situations.
Figures
Reference graph
Works this paper leans on
-
[1]
Sound event detection: A tutorial,
A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumb- ley, “Sound event detection: A tutorial,” IEEE Signal Processing Magazine, vol. 38, no. 5, pp. 67–83, 2021
work page 2021
-
[2]
Y . Zhang, R. Togneri, and D. Huang, “A unified loss function to tackle inter-class and intra-class data imbal- ance in sound event detection,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2024, pp. 996–1000
work page 2024
-
[3]
J. Wu, F. Yang, and W. Hu, “Unsupervised anomalous sound detection for industrial monitoring based on arcface classifier and gaussian mixture model,” Appl. Acoust., vol. 203, p. 109 188, 2023
work page 2023
-
[4]
W. Qiu, C. Quan, L. Zhu, et al. , “Heart sound abnor- mality detection from multi-institutional collaboration: Introducing a federated learning framework,” IEEE Trans. Biomed. Eng., vol. 71, no. 10, pp. 2802–2813, 2024
work page 2024
-
[5]
E.-L. Tan, F. A. Karnapi, L. J. Ng, K. Ooi, and W.-S. Gan, “Extracting urban sound information for residential areas in smart cities using an end-to-end iot system,” IEEE Internet Things J. , vol. 8, no. 18, pp. 14 308–14 321, 2021
work page 2021
-
[6]
A decade of dcase: Achievements, practices, evaluations and future challenges,
A. Mesaros, R. Serizel, T. Heittola, T. Virtanen, and M. D. Plumbley, “A decade of dcase: Achievements, practices, evaluations and future challenges,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2025, pp. 1–5
work page 2025
-
[7]
Ast-sed: An effective sound event detection method based on audio spectrogram transformer,
K. Li, Y . Song, L.-R. Dai, I. McLoughlin, X. Fang, and L. Liu, “Ast-sed: An effective sound event detection method based on audio spectrogram transformer,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2023, pp. 1–5
work page 2023
-
[8]
Diffsed: Sound event detection with denoising diffusion,
S. Bhosale, S. Nag, D. Kanojia, J. Deng, and X. Zhu, “Diffsed: Sound event detection with denoising diffusion,” in Proc. AAAI Conf. Artif. Intell. , vol. 38, 2024, pp. 792– 800
work page 2024
Show all 29 references
-
[9]
Fine-tune the pretrained atst model for sound event detection,
N. Shao, X. Li, and X. Li, “Fine-tune the pretrained atst model for sound event detection,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2024, pp. 911–915
2024
-
[10]
Ucil: An unsupervised class incremental learning approach for sound event detection,
Y . Xiao and R. Kumar Das, “Ucil: An unsupervised class incremental learning approach for sound event detection,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2025, pp. 1–5
2025
-
[11]
Full-frequency dynamic convolution: A physi- cal frequency-dependent convolution for sound event detection,
H. Yue, Z. Zhang, D. Mu, Y . Dang, J. Yin, and J. Tang, “Full-frequency dynamic convolution: A physi- cal frequency-dependent convolution for sound event detection,” in Proc. Int. Conf. Pattern Recognit. , 2025, pp. 260–272
2025
-
[12]
Sound activity- aware based cross-task collaborative training for semi- supervised sound event detection,
Y . Guan, J. Han, H. Song, et al. , “Sound activity- aware based cross-task collaborative training for semi- supervised sound event detection,” IEEE/ACM Trans. Audio Speech Lang. Process. , vol. 32, pp. 3947–3959, 2024
2024
-
[13]
Exploring text-queried sound event detection with audio source separation,
H. Yin, J. Bai, Y . Xiao, et al., “Exploring text-queried sound event detection with audio source separation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2025, pp. 1–5
2025
-
[14]
Multi-scale convolutional recurrent neural network with ensemble method for weakly labeled sound event detection,
Y . Guo, M. Xu, Z. Wu, J. Wu, and B. Su, “Multi-scale convolutional recurrent neural network with ensemble method for weakly labeled sound event detection,” in Proc. 8th Int. Conf. Affect. Comput. Intell. Interact. Workshops Demos, 2019, pp. 1–5
2019
-
[15]
A model ensemble approach for sound event localization and detection,
Q. Wang, H. Wu, Z. Jing, et al. , “A model ensemble approach for sound event localization and detection,” in Proc. 12th Int. Symp. Chinese Spoken Lang. Process. , 2021, pp. 1–5
2021
-
[16]
A sound event detection based on hybrid convolution neural network and random forest,
M. A. S. M. Afendi and M. Yusoff, “A sound event detection based on hybrid convolution neural network and random forest,” IAES Int. J. Artif. Intell. , vol. 11, no. 1, pp. 121–128, 2022
2022
-
[17]
A large multi-modal ensemble for sound event detection,
H. Dinkel, Z. Yan, Y . Wang, M. Song, J. Zhang, and W. Wang, “A large multi-modal ensemble for sound event detection,” IEEE AASP Challenge on DCASE 2022, Tech. Rep., 2022
2022
-
[18]
An ensemble of convolutional neural networks for sound event detection,
A. Mukhamadiyev, I. Khujayarov, D. Nabieva, and J. Cho, “An ensemble of convolutional neural networks for sound event detection,” Mathematics, vol. 13, no. 9, pp. 1502–1527, 2025
2025
-
[19]
Open-environment machine learning,
Z.-H. Zhou, “Open-environment machine learning,” Na- tional Science Review , vol. 9, no. 8, nwac123, 2022
2022
-
[20]
WildDESED: An LLM-powered dataset for wild domestic environment sound event detection system,
Y . Xiao and R. K. Das, “WildDESED: An LLM-powered dataset for wild domestic environment sound event detection system,” in Proc. Detection Classification Acoust. Scenes Events Workshop , 2024, pp. 196–200
2024
-
[21]
Mixstyle based domain generalization for sound event detec- tion with heterogeneous training data,
Y . Xiao, H. Yin, J. Bai, and R. K. Das, “Mixstyle based domain generalization for sound event detec- tion with heterogeneous training data,” arXiv preprint arXiv:2407.03654, 2024
2024 arXiv
-
[22]
A-crnn: A domain adaptation model for sound event detection,
W. Wei, H. Zhu, E. Benetos, and Y . Wang, “A-crnn: A domain adaptation model for sound event detection,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. , 2020, pp. 276–280
2020
-
[23]
Robust audio deepfake detection using ensemble confidence calibration,
C. Y . Kwok, D. -T. Truong, and J. Q. Yip, “Robust audio deepfake detection using ensemble confidence calibration,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2025, pp. 1–5
2025
-
[24]
On calibration of modern neural networks,
C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Int. Conf. Machine Learning, 2017, pp. 1321–1330
2017
-
[25]
Energy-based open-world uncertainty modeling for confidence calibration,
Y . Wang, B. Li, T. Che, K. Zhou, Z. Liu, and D. Li, “Energy-based open-world uncertainty modeling for confidence calibration,” in Proc. IEEE/CVF Int. Conf. Computer Vision, 2021, pp. 9302–9311
2021
-
[26]
Object detectors in the open environment: Challenges, solutions, and outlook,
S. Liang, W. Wang, R. Chen, et al., “Object detectors in the open environment: Challenges, solutions, and outlook,” arXiv preprint arXiv:2403.16271 , 2024
2024 arXiv
-
[27]
Tut database for acoustic scene classification and sound event de- tection,
A. Mesaros, T. Heittola, and T. Virtanen, “Tut database for acoustic scene classification and sound event de- tection,” in 24th Eur. Signal Process. Conf. , 2016, pp. 1128–1132. 5
2016
-
[28]
Dcase 2017 challenge setup: Tasks, datasets and baseline system,
A. Mesaros, T. Heittola, A. Diment, et al., “Dcase 2017 challenge setup: Tasks, datasets and baseline system,” in Proc. Detection Classification Acoust. Scenes Events Workshop, 2017
2017
-
[29]
Metrics for polyphonic sound event detection,
A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,” Appl. Sci. , vol. 6, no. 6, pp. 162–178, 2016. 6
2016
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.