REVIEW 5 minor 38 references
Technical Report for MERL's Real-TSE Challenge Submission
T0 review · 0 major / 5 minor · reviewed 2026-07-13 · grok-4.5
Pith's one-line read Careful multi-stage data preparation, not a new architecture, won first place in real-world target speech extraction; DNSMOS and speaker similarity can be adversarially maxed without harming speech recognition or VAD scores.
desk verdict Solid first-place Real-TSE report: data curriculum + clean metric-fragility demo, not a new architecture. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Four-stage curriculum that ends with real far-field mixtures whose supervised targets are produced by close-talk enhancement, max-peak synchronization, causal Wiener projection onto the distant channel, and a second enhancement pass; metric-aware losses and a subsequent per-utterance adversarial attack that regularizes spectrogram magnitude while maximizing DNSMOS and cosine speaker similarity.
What would settle it
Train an identical architecture on the same real far-field mixtures but replace the Wiener-projected pseudo-targets with either pure silence or with unprocessed close-talk signals; if token error rate and VAD F1 on the Real-TSE evaluation set collapse, the pseudo-target quality claim is falsified.
Extended reading notes
Core claim
A baseline Band-split RNN with a larger speaker encoder, trained through four progressive stages that move from fully overlapped clean mixtures to real far-field multi-talker conversations whose targets are pseudo-labels obtained from processed close-talk microphones, achieves first place on the Real-TSE Challenge second track; the same work also demonstrates that DNSMOS and speaker-similarity scores can be pushed to extreme values by imperceptible adversarial noise while token error rate and VAD-based F1 remain essentially unchanged.
Load-bearing premise
The multi-step close-talk-to-far-field projection pipeline produces pseudo-targets accurate and unbiased enough to serve as reliable supervised labels for real multi-talker mixtures.
Editorial extensions
If this is right
- Future Real-TSE or similar challenges can expect large gains from data-cleaning and curriculum design even when model architecture is held fixed.
- Leaderboards that include DNSMOS or speaker-similarity scores can be gamed by small adversarial perturbations that leave ASR and VAD metrics intact.
- Challenge organizers are given concrete evidence to drop or replace those two metrics when computing official rankings.
- Pseudo-label pipelines that project close-talk onto far-field channels become a reusable recipe for adapting separation models to other real multi-microphone corpora.
Reading between the lines
- Any non-intrusive neural quality metric that is differentiable and has been seen during training is likely to admit similar low-energy adversarial attacks, so the problem is not limited to DNSMOS.
- Once pseudo-target quality is accepted as the bottleneck, the next practical research question becomes how little close-talk supervision is still sufficient for the projection stage.
- The same curriculum-plus-projection recipe could be tested on streaming or multi-channel TSE without changing the core network, offering a low-risk path to on-device systems.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This technical report describes MERL’s first-place submission to the Real-TSE Challenge. The authors retain a strengthened baseline BSRNN with multi-level speaker conditioning (EcapaTDNN-1024, 10 blocks) and concentrate on data preparation and a four-stage curriculum: (1) fully-overlapped Libri2Mix-style pre-training, (2) noisy/reverberant simulated conversations, (3) on-the-fly far-field mixtures from single-talker segments of CHiME-6/AMI/AISHELL, and (4) real multi-talker mixtures supervised by pseudo-targets obtained via close-talk enhancement, max-peak synchronization, causal Wiener projection and a second enhancement pass. Table I documents monotonic gains in TER and F1 across stages; a pure-reconstruction ablation matches the final TER/DNSMOS. An additional per-utterance adversarial attack (Algorithm 1, Table II) drives DNSMOS and speaker-similarity scores to extremes while leaving TER and VAD-based F1 unchanged, illustrating metric fragility.
Significance. If the reported ranking and ablation hold, the work supplies concrete evidence that careful multi-source data cleaning, heavy mixture/enrollment augmentation and staged adaptation to real far-field conditions can outperform architectural novelty on a difficult real-world TSE benchmark. The controlled metric attack cleanly isolates the brittleness of DNSMOS and speaker similarity (Goodhart’s law), offering a practical warning for challenge organizers. Strengths that raise the contribution above a pure system description include the external leaderboard validation, the pure-reconstruction ablation in Table I, and the fully specified attack procedure that preserves the non-attacked metrics.
minor comments (5)
- Throughout the manuscript (abstract, §I, §VII) the challenge name appears inconsistently as “Real-TSE”, “RealTSE” and “Real-T”. Standardize to one form.
- Algorithm 1 contains incomplete expressions (“loss = loss.pow().mean()”) and uses undefined helpers (stft, dnsmos, spksim). A short clarifying sentence or corrected pseudocode would improve reproducibility.
- §III.A–C repeatedly write “ClearerV oice”, “V oxCeleb2”, “V AD”. These appear to be PDF-extraction artifacts; restore the correct spellings (ClearerVoice, VoxCeleb2, VAD).
- Eq. (5) weights are given without justification or sensitivity analysis. A one-sentence remark on how the coefficients 10 / 0.01 were chosen would help readers.
- Table I caption uses “SPKFOR SPEAKER” (missing space) and the final pure-reconstruction row is labeled only by a star; a clearer legend would avoid ambiguity.
Circularity Check
No load-bearing circularity; mild self-training bootstrap on pseudo-targets is secondary and does not force the external ranking or attack results.
-
other
[Section III.C (Real Far-field Mixture Training) and Section IV.D (fourth training stage)]
"Afterwards, we use the best-performing TSE model trained on far-field mixtures to reduce cross-talk on CHiME-6 and AMI close-talk microphones. ... Lastly, the fourth stage contains real noisy mixture training against targets that were extracted by our previous TSE model."
A prior version of the same model family produces the pseudo-targets used to train the submitted model. This is iterative self-training rather than a definitional identity, and evaluation (TER, F1, external ranking) remains independent of the pseudo-targets; it is therefore only a mild, non-load-bearing circularity.
full rationale
This is an empirical challenge technical report, not a first-principles derivation. The central claims (first-place ranking on the Real-TSE second track; DNSMOS/spk-sim fragility under adversarial attack) rest on external leaderboard evaluation and on a controlled post-hoc per-utterance attack (Algorithm 1, Table II) that leaves TER and VAD-F1 unchanged while driving the attacked metrics to extremes. Table I further shows that the progressive TER/F1 gains across the four curriculum stages, and the pure-reconstruction ablation (last row), do not require the optional metric-aware losses. The only mild circular element is the fourth-stage use of a previous TSE model to clean close-talk signals that then become pseudo-targets for the next model; this is ordinary iterative self-training / bootstrapping, not a definitional reduction of a claimed prediction to its own inputs, and the final metrics remain independent of those pseudo-targets. No self-definitional equations, fitted-parameter-as-prediction, load-bearing self-citation uniqueness theorems, or renamed known results appear. Score 1 reflects only that secondary bootstrap; the paper is otherwise self-contained against external benchmarks.
Assumptions & free parameters
free parameters (8)
- speaker-similarity filter threshold =
0.85
- DNSMOS pre-filter threshold =
3.0
- WER filtering thresholds =
30 % / 50 %
- loss weights (Lstft, LMOS, Lspk) =
10 / 0.01 / 1
- RIR and noise application probability =
0.8
- non-target enrollment probability =
0.05
- learning rates and schedule =
5e-4 / 1e-4
- model capacity knobs =
10 blocks / 1024-dim
assumptions (4)
- domain assumption Processed close-talk signals after enhancement, synchronization and causal Wiener projection constitute valid supervised targets for far-field multi-talker mixtures.
- domain assumption Curriculum learning from fully-overlapped synthetic data through simulated conversations to real far-field mixtures improves final performance (Bengio et al., 2009).
- domain assumption Token error rate and VAD-based F1 are reliable proxies for extraction quality that cannot be gamed by the same adversarial perturbations that affect DNSMOS and speaker similarity.
- standard math Standard optimizers (AdamW), STFT magnitude losses and speaker-embedding cosine losses behave as expected for TSE.
Cite this review
Pith. "Pith review of Technical Report for MERL's Real-TSE Challenge Submission." pith.science (2026). https://pith.science/paper/MXJIEVZ4
@misc{pith2026260709043,
author = {Pith},
title = {Pith review of: Technical Report for MERL's Real-TSE Challenge Submission},
year = {2026},
howpublished = {\url{https://pith.science/paper/MXJIEVZ4}},
note = {Machine review of arXiv:2607.09043}
}
read the original abstract
Target speech extraction (TSE) has largely been dominated by neural network-based approaches trained and evaluated on synthetic fully overlapped data. The Real-TSE Challenge aims to advance performance on real-world far-field noisy and reverberant recordings. This technical report describes MERL's submission to the Real-TSE Challenge. Rather than proposing a novel model architecture, we built upon the baseline model and focused primarily on data preparation and cleaning. Our system was trained in four stages, beginning with pre-training on fully overlapped mixtures and simulated multi-talker conversations with noise and reverberation applied to both the mixture and the enrollment utterances. We then adapted the model to real-world conditions using noisy far-field recordings with pseudo-targets derived from processed close-talk microphone signals. Our submission achieved first place in the second track, demonstrating the critical importance of high-quality data preparation. Furthermore, we observed that DNSMOS and speaker similarity are susceptible to over-optimization, motivating an investigation of their robustness using adversarial attacks. The results show that both metrics can be driven to extreme values without degrading the token error rate or the VAD-based F1 score.
Reference graph
Works this paper leans on
-
[1]
LibriMix: An open-source dataset for generalizable speech separation,
J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, “LibriMix: An open-source dataset for generalizable speech separation,” arXiv preprint arXiv:2005.11262, 2020
arXiv 2005
-
[2]
CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,
S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V . Manohar, D. Povey, D. Raj, D. Snyder, A. S. Subrama- nian, J. Trmal, B. B. Yair, C. Boeddeker, Z. Ni, Y . Fujita, S. Horiguchi, N. Kanda, T. Yoshioka, and N. Ryant, “CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,” inProc. 6th Inter...
2020
-
[3]
DiPCo – dinner party corpus,
M. Van Segbroeck, A. Zaid, K. Kutsenko, C. Huerta, T. Nguyen, X. Luo, B. Hoffmeister, J. Trmal, M. Omologo, and R. Maas, “DiPCo – dinner party corpus,” inProc. Interspeech, 2020, pp. 434–436
2020
-
[4]
REAL-T: Real conversational mixtures for target speaker extraction,
S. Li, S. Wang, J. Han, K. Zhang, W. Wang, and H. Li, “REAL-T: Real conversational mixtures for target speaker extraction,” inProc. Interspeech, 2025, pp. 1923–1927
2025
-
[5]
Multi-level speaker representation for target speaker extraction,
K. Zhang, J. Li, S. Wang, Y . Wei, Y . Wang, Y . Wang, and H. Li, “Multi-level speaker representation for target speaker extraction,” in Proc. ICASSP, 2025
2025
-
[6]
ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,
B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” inProc. Interspeech, 2020, pp. 3830–3834
2020
-
[7]
LibriSpeech: An ASR corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” inProc. ICASSP, 2015, pp. 5206–5210
2015
-
[8]
V oxCeleb2: Deep speaker recognition,
J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep speaker recognition,” inProc. Interspeech, 2018, pp. 1086–1090
2018
Show all 38 references
-
[9]
Emilia: A large- scale, extensive, multilingual, and diverse dataset for speech generation,
H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y . Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: A large- scale, extensive, multilingual, and diverse dataset for speech generation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 33, pp. 40...
2025
-
[10]
CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),
J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),” 2019
2019
-
[11]
EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation,
J. Richter, Y .-C. Wu, S. Krenn, S. Welker, B. Lay, S. Watanabe, A. Richard, and T. Gerkmann, “EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation,” in Proc. Interspeech, 2024, pp. 4873–4877
2024
-
[12]
ClearerV oice-Studio: Bridging advanced speech processing research and practical deployment,
S. Zhao, Z. Pan, and B. Ma, “ClearerV oice-Studio: Bridging advanced speech processing research and practical deployment,” inProc. Inter- speech, 2025, pp. 2980–2984
2025
-
[13]
MossFormer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augmented joint self-attentions,
S. Zhao and B. Ma, “MossFormer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augmented joint self-attentions,” inProc. ICASSP, 2023
2023
-
[14]
WeSpeaker: A research and production oriented speaker embedding learning toolkit,
H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y . Deng, and Y . Qian, “WeSpeaker: A research and production oriented speaker embedding learning toolkit,” inProc. ICASSP, 2023
2023
-
[15]
Montreal forced aligner: Trainable text-speech alignment using Kaldi,
M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using Kaldi,” inProc. Interspeech, 2017, pp. 498–502
2017
-
[16]
ICASSP 2022 deep noise suppression challenge,
H. Dubey, V . Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper, and R. Aichner, “ICASSP 2022 deep noise suppression challenge,” inProc. ICASSP, 2022, pp. 9271–9275
2022
-
[17]
Pyroomacoustics: A python package for audio room simulation and array processing algorithms,
R. Scheibler, E. Bezzam, and I. Dokmani ´c, “Pyroomacoustics: A python package for audio room simulation and array processing algorithms,” in Proc. ICASSP, 2018, pp. 351–355
2018
-
[18]
The third CHiME speech separation and recognition challenge: Dataset, task and base- lines,
J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third CHiME speech separation and recognition challenge: Dataset, task and base- lines,” inProc. ASRU, Dec. 2015, pp. 504–511
2015
-
[19]
The diverse environments multi- channel acoustic noise database (DEMAND): A database of multichan- nel environmental noise recordings,
J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi- channel acoustic noise database (DEMAND): A database of multichan- nel environmental noise recordings,” inProceedings of Meetings on Acoustics, vol. 19, no. 1. Acoustical Society of America, 2013, p. 035081
2013
-
[20]
FMA: A dataset for music analysis,
M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” inProc. ISMIR, 2017, pp. 316–323
2017
-
[21]
FSD50K: An open dataset of human-labeled sound events,
E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An open dataset of human-labeled sound events,”IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 30, pp. 829–852, 2022
2022
-
[22]
MUSAN: A music, speech, and noise corpus,
D. Snyder, G. Chen, and D. Povey, “MUSAN: A music, speech, and noise corpus,”arXiv preprint arXiv:1510.08484, 2015
2015 arXiv
-
[23]
WHAM!: Extending speech separation to noisy environments,
G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux, “WHAM!: Extending speech separation to noisy environments,” inInterspeech 2019, 2019
2019
-
[24]
URGENT challenge: Universality, robustness, and generalizability for speech en- hancement,
W. Zhang, R. Scheibler, K. Saijo, S. Cornell, C. Li, Z. Ni, J. Pirkl- bauer, M. Sach, S. Watanabe, T. Fingscheidt, and Y . Qian, “URGENT challenge: Universality, robustness, and generalizability for speech en- hancement,” inProc. Interspeech, 2024, pp. 4868–4872
2024
-
[25]
The AMI meeting corpus: A pre-announcement,
J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V . Karaiskos, W. Kraaij, M. Kronenthalet al., “The AMI meeting corpus: A pre-announcement,” inInternational workshop on machine learning for multimodal interaction. Springer, 2005, pp. 28– 39
2005
-
[26]
AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,
Y . Fu, L. Cheng, S. Lv, Y . Jv, Y . Kong, Z. Chen, Y . Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, “AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,” inProc. Interspeech, 2021, pp. 3665–3669
2021
-
[27]
AISHELL-5: The first open-source in-car multi-channel multi-speaker speech dataset for automatic speech diarization and recognition,
Y . Dai, H. Wang, X. Li, Z. Zhang, S. Wang, L. Xie, X. Xu, H. Guo, S. Zhang, H. Bu, and W. Chen, “AISHELL-5: The first open-source in-car multi-channel multi-speaker speech dataset for automatic speech diarization and recognition,” inProc. Interspeech, 2025, pp. 5493–5497
2025
-
[28]
Front-end processing for the CHiME-5 dinner party scenario,
C. Boeddecker, J. Heitkaemper, J. Schmalenstroeer, L. Drude, J. Hey- mann, and R. Haeb-Umbach, “Front-end processing for the CHiME-5 dinner party scenario,” inProc. 5th International Workshop on Speech Processing in Everyday Environments (CHiME 2018), 2018, pp. 35–40
2018
-
[29]
Gen- erating training targets for real-world speech enhancement via close- to-distant microphone projection,
T. Nakatani, R. Ikeshita, N. Kamo, M. Delcroix, and S. Araki, “Gen- erating training targets for real-world speech enhancement via close- to-distant microphone projection,” inProc. ICASSP, 2026, pp. 18 912– 18 916
2026
-
[30]
Canary-1B-V2 & Parakeet- TDT-0.6B-V3: Efficient and high-performance models for multilingual ASR and AST,
M. Sekoyan, N. R. Koluguri, N. Tadevosyan, P. Zelasko, T. Bartley, N. Karpov, J. Balam, and B. Ginsburg, “Canary-1B-V2 & Parakeet- TDT-0.6B-V3: Efficient and high-performance models for multilingual ASR and AST,”arXiv preprint arXiv:2509.14128, 2025
2025
-
[31]
Curriculum learning,
Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProc. ICML, ser. ACM International Conference Proceeding Series, vol. 382. ACM, 2009, pp. 41–48
2009
-
[32]
Mind the gap: Impact of synthetic conversational data on multi-talker ASR and speaker diarization,
A. Polok, I. Medennikov, J. ˇCernock`y, S. Watanabe, L. Burget, and S. Cornell, “Mind the gap: Impact of synthetic conversational data on multi-talker ASR and speaker diarization,”arXiv preprint arXiv:2605.15442, 2026
2026 arXiv
-
[33]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inProc. ICLR, 2019
2019
-
[34]
SGDR: Stochastic gradient descent with warm restarts,
——, “SGDR: Stochastic gradient descent with warm restarts,” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/forum?id=Skq89Scxx
2017
-
[35]
The text-to-speech in the wild (TITW) database,
J.-W. Jung, W. Zhang, S. Maiti, Y . Wu, X. Wang, J.-H. Kim, Y . Mat- sunaga, S. Um, J. Tian, H.-J. Shim, N. Evans, J. S. Chung, S. Takamichi, and S. Watanabe, “The text-to-speech in the wild (TITW) database,” in Interspeech 2025. ISCA: ISCA, Aug. 2025, pp. 4798–4802
2025
-
[36]
Multilayer feedforward networks are universal approximators,
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,”Neural Networks, vol. 2, no. 5, pp. 359–366, 1989. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/0893608089900208
1989
-
[37]
Beyond waveform robustness: Robust feature-vocoder adversarial attacks on automatic speech recognition,
Y . Liao, Z. Zhang, Z. Sun, Y . Sun, X. Zheng, and X. He, “Beyond waveform robustness: Robust feature-vocoder adversarial attacks on automatic speech recognition,”arXiv preprint arXiv:2606.05678, 2026
2026 arXiv
-
[38]
Attacking UTMOS: Probing the robustness of a speech quality assessment model,
W.-C. Huang and T. Toda, “Attacking UTMOS: Probing the robustness of a speech quality assessment model,” 2026. [Online]. Available: https://arxiv.org/abs/2606.31105
2026 arXiv
Reviewed July 13, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.