REVIEW 3 major objections 5 minor 1 cited by
URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Trained on 292 English utterances, the pairwise-comparison model URGENT-PK ranks unseen speech-enhancement systems more accurately than DNSMOS and UTMOS across four languages and an unseen acoustic domain.
desk verdict A useful pairwise ranking model for speech enhancement evaluation with real cross-language evidence, but the headline English result is partly compromised by test-set tuning of the data-cleaning threshold; the multilingual results are cleaner. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pairing of an utterance-level pairwise model with a system-level Enumerating-Comparing-Scoring (ECS) algorithm. The pairwise model takes two enhanced versions of the same noisy utterance, encodes them separately through either a log-mel spectrogram encoder or a UTMOS-based encoder (frozen or fine-tuned), concatenates the temporal embeddings along a new channel dimension, and feeds the fused representation to a modified ResNet34. The network outputs a comparative score in $[0,1]$ and estimated MOS values for both inputs. The ECS algorithm then enumerates all $K(K-1)/2$ system pairs, runs the pairwise model over all $M$ utterances for each pair, and accumulates one point to the winner (binary scoring) or fractional scores proportional to the comparative score (non-binary scoring), with the final ranking given by the accumulated scores. The paper emphasizes the explicit comparison module as the architectural departure from RankNet and MicRank, which encode ranking relationships only through their loss functions.
What would settle it
Re-run the full training and checkpoint-selection pipeline on several different random 8-utterance validation splits drawn from the 292 urgent24 English training utterances, and measure the spread of test-set correlations on urgent25 English; if the best URGENT-PK variant's Kendall rank correlation varies by more than roughly 0.05 across splits, the reported margins over DNSMOS and UTMOS are an artifact of the single validation split rather than a stable property of the method.
Extended reading notes
Core claim
URGENT-PK's central discovery is that a pairwise comparison paradigm extracts far more supervision from scarce MOS-labeled data than direct MOS regression, because every pair of systems processing the same noisy input yields a comparison and the $K(K-1)/2$ system pairs multiply the effective training signal. The utterance-level model encodes two homologous speech samples (the same noisy utterance enhanced by two different systems), concatenates the embeddings, and passes them through a ResNet34-based comparison module that outputs a comparative score in $[0,1]$ along with estimated MOS for both inputs; training minimizes a binary cross-entropy loss on the comparison against the sign of the MOS difference plus an MSE loss on the MOS estimates. At inference, the ECS algorithm enumerates all system pairs, scores every utterance pair, and accumulates either binary or fractional points per system to produce the final ranking. On urgent25 English, the best URGENT-PK variant (with fine-tuned UTMOS encoder) reaches a Kendall rank correlation of 0.879 and a Spearman correlation of 0.972, and even the log-mel variant trained from scratch outperforms fine-tuned UTMOS, a pattern the authors say holds on multilingual and cross-domain test sets. They interpret these results as evidence that a simple, data-efficient comparison model can act as a perceptually aligned ranking engine for SE competitions.
Load-bearing premise
The load-bearing premise is that the training labels and model selection are trustworthy: pairwise labels are created by thresholding the sign of an 8-listener average MOS difference at 0.3, and every hyperparameter and the final checkpoint are chosen using a validation set of only 8 utterances, so the claimed cross-domain generalization depends on that tiny split being representative and on the MOS-difference proxy faithfully matching human pairwise preference.
Editorial extensions
If this is right
- Speech enhancement challenges could replace or pre-filter full MOS listening tests with a pairwise comparison model, since a few hundred labeled utterances suffice to train a competitive ranker.
- The pairwise objective multiplies scarce data: with 22 systems and 150 utterances per system, each utterance yields 231 system-pair comparisons, so the effective training-set size is far larger than the raw utterance count.
- The same model can also act as a MOS predictor via replication or noisy-speech prompting, giving it a fallback when absolute scores are required.
- Because the log-mel variant trained from scratch is competitive, pre-trained self-supervised encoders are not necessary for good ranking performance, lowering the computational and data barrier to deployment.
- The reported transfer across Chinese, Japanese, and German suggests that relative quality judgments generalize across languages better than absolute MOS scales, which could help multilingual speech quality assessment.
Reading between the lines
- Because the labels are derived from averaged MOS differences rather than direct pairwise human judgments, the model inherits any bias in the original listening test; collecting true A/B preference labels on the same utterances might yield a cleaner training signal and likely shift the optimal MOS-difference threshold.
- The ECS aggregation treats every utterance pair equally; weighting comparisons by confidence or by the size of the comparative margin could improve ranking stability when systems are close in quality.
- The pairwise model only compares systems on the same noisy input, so it cannot assess cross-condition consistency; a test that mixes noisy conditions within a pair would probe whether the learned comparison transfers to conditions not seen in training.
- A natural extension is to apply the same pairwise-comparison-plus-aggregation scheme to other quality dimensions such as intelligibility, naturalness, or speaker similarity, where absolute scales are equally noisy and relative preferences may be more consistent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes URGENT-PK, a pairwise comparison model for ranking speech enhancement (SE) systems. The utterance-level model takes two homologous enhanced speech samples produced from the same noisy input, encodes them with either a log-mel front end or a UTMOS-based encoder, and uses a ResNet34 comparison module to output a comparative score plus two MOS estimates. The system-level Enumerating-Comparing-Scoring (ECS) algorithm accumulates wins or scores over all system pairs. The model is trained on the urgent24en dataset with a multi-task objective combining pairwise BCE loss and MOS MSE loss, and is evaluated on urgent25en, urgent25zh/ja/de, and CHiME-7 UDASE by computing system-level KRCC, SRCC, and LCC against oracle MOS. The authors report that URGENT-PK variants outperform DNSMOS, UTMOS, fine-tuned UTMOS, and UTMOS with ECS on most test sets, despite a simple architecture and limited training data.
Significance. If the empirical claims are robust, the paper makes a useful contribution: it shows that a pairwise ranking paradigm can exploit limited paired MOS data more efficiently than absolute MOS prediction, and that system-level rankings can be obtained from an interpretable voting algorithm. The multi-task design (pairwise score plus predicted MOS) is a reasonable way to inject perceptual knowledge, and the evaluation on multiple languages and a different challenge (CHiME-7 UDASE) is a genuine strength. The authors also honestly report the one clear failure case (UG-PKmel on urgent25zh) and provide an ablation of the data-cleaning threshold. However, the evaluation has load-bearing robustness issues: the main in-domain test set was used to choose a hyperparameter, the validation set has only 8 utterances, and no confidence intervals or significance tests are reported. These issues do not necessarily invalidate the central claim, but they need to be addressed before the reported superiority can be taken at face value.
major comments (3)
- [Section III-F4 / Figure 2 / Table III] The data-cleaning threshold δ=0.3 is selected by an ablation performed on urgent25en (Section III-F4, Figure 2), and the same urgent25en set is the test set for the headline results in Table III. This means the in-domain English comparison is not fully held out: the threshold was chosen by looking at exactly the systems and utterances whose correlations are then reported as the main evidence. The margins over UTMOSft (e.g., KRCC 0.879 vs. 0.814) could therefore be partially inflated by test-set tuning. Please re-select δ using the validation split or an inner cross-validation on the training split, and re-run Table III with that chosen value; if the conclusions change, the abstract's claim about 'multiple open test sets' should be rephrased to emphasize the multilingual and CHiME-7 results, which are not affected by this particular tuning step.
- [Section III-E / Tables II and III] The validation set used for checkpoint selection contains only 8 utterances, and no confidence intervals, bootstrap estimates, or significance tests are reported anywhere in the paper. System-level correlations on 22 systems (urgent25) and especially on 5 systems (CHiME-7) have coarse resolution: for CHiME-7, a single system reordering can change KRCC or SRCC by a large discrete step, so differences such as 0.400 vs. 0.200 or 0.500 vs. 0.300 may easily be within sampling noise. Please report confidence intervals (e.g., bootstrap over systems or over utterances) for the correlation values and, where meaningful, a paired significance test between URGENT-PK and the strongest baseline. Without this, the reader cannot tell whether the observed gaps are systematic or accidental.
- [Section III-B and Section III-C] The loss weights α=β=0.5 and the number of mel filters (120) are presented as fixed choices without ablations or sensitivity analysis, and the data-cleaning threshold δ is the only hyperparameter studied. Since the paper's argument is that the pairwise paradigm is data-efficient and robust, it would strengthen the paper to show that the main conclusions are not sensitive to reasonable variations of α, β, and the mel filter count. At minimum, please state whether these values were chosen on the 8-utterance validation set, and if so, acknowledge the instability risk of tuning on such a small set.
minor comments (5)
- [Figure 2] The axis labels and bar values in Figure 2 are rendered as garbled unicode replacement sequences (e.g., '/uni00000013/uni00000011/...'), making the actual threshold values and correlation values unreadable. Please replace the figure with a clean version and, ideally, also provide the numeric values in a table.
- [Table IV] The row label 'MOScp' is not defined. The text describes a comparison between the pairwise model and 'MOS comparison', but it is unclear whether MOScp denotes the accuracy of comparing the oracle MOS values, the predicted MOS values, or something else. Please define the row explicitly.
- [Section II-A] There is a typo in 'fundamentally differeserent' and later in Section II-B2 the text says 'In the Binary Scoring (BS) strategy' when describing the non-binary strategy; these should be corrected.
- [Section III-B] In the range notation '[0,0.4)' and similar, there are missing spaces after commas; also the phrase 'no more confusing' should likely be 'no longer confusing'. These small language and formatting issues should be fixed in revision.
- [General] The paper does not state whether the URGENT-PK training code, model checkpoints, or inference scripts will be released. Given the reproducibility-oriented evaluation on open challenge data, please add an availability statement or explain why the artifacts are not released.
Circularity Check
No definitional circularity; one fitted test-set threshold affects the in-domain English comparison, but the central claim is otherwise externally validated.
-
fitted input called prediction
[Section III-B; Section III-F4 / Fig. 2; Table III]
"In this paper, we set the score difference threshold to δ = 0.3... Ablation studies in Section III-F4 demonstrate the rationality of this threshold setting. ... Fig. 2. Ablation study of the MOS difference threshold δ in data cleaning. Models are trained on urgent24 en and tested on urgent25 en."
The ablation used to justify δ=0.3 is run on urgent25en, the same set on which Table III reports headline correlations. Selecting δ by peaking Fig. 2 on urgent25en and then reporting UG-PK performance on urgent25en is a fitted hyperparameter evaluated on the same data; the in-domain margin over UTMOSft is partly produced by test-set selection. This is not a definitional equivalence: rankings still depend on trained weights and other fixed hyperparameters. Multilingual and CHiME-7 results in Table II were not used to choose δ, so the central generalization claim retains independent support.
full rationale
The pairwise training target int(MOS1>MOS2) is derived from the same external MOS labels used for evaluation, but that is standard supervised learning, not circularity: the model is trained on urgent24en and evaluated on urgent25en, CHiME-7, and multilingual sets with oracle average MOS. The UTMOS encoder is borrowed from an external challenge system and is also used as a baseline; this creates a transfer-learning advantage but no self-referential derivation. Self-citations [14], [15] are shared challenge datasets, not load-bearing arguments. There is no uniqueness theorem or ansatz smuggled via citation. The only notable issue is the δ=0.3 data-cleaning threshold, selected by an ablation performed on the urgent25en test set; this is a test-set tuning caveat rather than a by-construction circularity. Because Table II's out-of-domain results and CHiME-7 results are not affected by this selection, the central claim of superior ranking generalization is not forced by the paper's own definitions. Overall circularity is minimal.
Assumptions & free parameters
free parameters (3)
- MOS difference threshold delta =
0.3
- Multi-task loss weights alpha, beta =
0.5, 0.5
- Number of mel filters =
120
assumptions (6)
- standard math BCE loss on pairwise labels and MSE loss on MOS outputs are appropriate training objectives.
- domain assumption The average MOS over 8 listeners is a reliable gold standard for speech quality.
- domain assumption Pairs with MOS difference above delta=0.3 are perceptually distinguishable and informative; pairs below are not used.
- domain assumption Accumulating pairwise comparison scores over all system pairs preserves the true system-level quality order.
- domain assumption The unprocessed noisy speech can be treated as one of the systems to be ranked.
- domain assumption UTMOS latent features provide a sufficient representation for quality comparison even when frozen.
Cite this review
Pith. "Pith review of URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition." pith.science (2026). https://pith.science/paper/WSAEPZ3X
@misc{pith2026250623874,
author = {Pith},
title = {Pith review of: URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition},
year = {2026},
howpublished = {\url{https://pith.science/paper/WSAEPZ3X}},
note = {Machine review of arXiv:2506.23874}
}
read the original abstract
The Mean Opinion Score (MOS) is fundamental to speech quality assessment. However, its acquisition requires significant human annotation. Although deep neural network approaches, such as DNSMOS and UTMOS, have been developed to predict MOS to avoid this issue, they often suffer from insufficient training data. Recognizing that the comparison of speech enhancement (SE) systems prioritizes a reliable system comparison over absolute scores, we propose URGENT-PK, a novel ranking approach leveraging pairwise comparisons. URGENT-PK takes homologous enhanced speech pairs as input to predict relative quality rankings. This pairwise paradigm efficiently utilizes limited training data, as all pairwise permutations of multiple systems constitute a training instance. Experiments across multiple open test sets demonstrate URGENT-PK's superior system-level ranking performance over state-of-the-art baselines, despite its simple network architecture and limited training data.
Figures
Forward citations
Cited by 1 Pith paper
-
Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment
A multi-metric learned quality model (Uni-VERSA-Ext) is used as a differentiable training loss for speech enhancement, with a regularization term to prevent adversarial exploitation.
Reference graph
Works this paper leans on
-
[1]
DNSMOS: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP, pages 6493–6497. IEEE, 2021
work page 2021
-
[2]
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP, pages 886–890. IEEE, 2022
work page 2022
-
[3]
UTMOS: UTokyo-SaruLab system for V oiceMOS challenge 2022
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinno- suke Takamichi, and Hiroshi Saruwatari. UTMOS: UTokyo-SaruLab system for V oiceMOS challenge 2022. In Interspeech, pages 4521– 4525, 2022
work page 2022
-
[4]
Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. The t05 system for the VoiceMOS Challenge 2024: Transfer learning from deep image classifier to naturalness MOS prediction of high-quality synthetic speech. In IEEE Spoken Language Technology Workshop (SLT) , 2024
work page 2024
-
[5]
Ky- otoMOS: An automatic MOS scoring system for speech synthesis
Wangjin Zhou, Zhengdong Yang, Sheng Li, and Chenhui Chu. Ky- otoMOS: An automatic MOS scoring system for speech synthesis. In Proceedings of the 5th ACM International Conference on Multimedia in Asia Workshops, pages 1–3, 2023
work page 2023
-
[6]
Generalization ability of MOS prediction networks
Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi. Generalization ability of MOS prediction networks. In ICASSP, pages 8442–8446, 2022
work page 2022
-
[7]
LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech
Wen-Chin Huang, Erica Cooper, Junichi Yamagishi, and Tomoki Toda. LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech. In ICASSP, pages 896–900, 2022
work page 2022
-
[8]
ITU-T recommendation P.808, subjective evaluation of speech quality with a crowdsourcing approach
ITU-T. ITU-T recommendation P.808, subjective evaluation of speech quality with a crowdsourcing approach. ITU-T, June 2018
work page 2018
Show all 27 references
-
[9]
An open source implementation of ITU- T recommendation P.808 with validation
Babak Naderi and Ross Cutler. An open source implementation of ITU- T recommendation P.808 with validation. In Interspeech, pages 2862– 2866, 2020
2020
-
[10]
Learning to rank microphones for distant speech recognition
Samuele Cornell, Alessio Brutti, Marco Matassoni, and Stefano Squar- tini. Learning to rank microphones for distant speech recognition. In Interspeech, pages 3855–3859, 2021
2021
-
[11]
A review of deep learning based speech synthesis
Yishuang Ning, Sheng He, Zhiyong Wu, Chunxiao Xing, and Liang- Jie Zhang. A review of deep learning based speech synthesis. Applied Sciences, 9(19):4050, 2019
2019
-
[12]
An overview of voice conversion and its challenges: From statistical modeling to deep learning
Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. An overview of voice conversion and its challenges: From statistical modeling to deep learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:132–157, 2020
2020
-
[13]
Learning to rank for information retrieval
Tie-Yan Liu et al. Learning to rank for information retrieval. F oundations and Trends® in Information Retrieval , 3(3):225–331, 2009
2009
-
[14]
URGENT challenge: Universality, robustness, and generalizability for speech enhancement
Wangyou Zhang, Robin Scheibler, Kohei Saijo, Samuele Cornell, Chenda Li, Zhaoheng Ni, Jan Pirklbauer, Marvin Sach, Shinji Watanabe, Tim Fingscheidt, and Yanmin Qian. URGENT challenge: Universality, robustness, and generalizability for speech enhancement. In Interspeech, pages ...
2024
-
[15]
Interspeech 2025 URGENT speech enhancement challenge
Kohei Saijo, Wangyou Zhang, Samuele Cornell, Robin Scheibler, Chenda Li, Zhaoheng Ni, Anurag Kumar, Marvin Sach, Yihui Fu, Wei Wang, Tim Fingscheidt, and Shinji Watanabe. Interspeech 2025 URGENT speech enhancement challenge. Accepted by Interspeech , 2025
2025
-
[16]
CHiME-7 UDASE evaluation data, February 2024
Simon Leglaive, Matthieu Fraticelli, Hend ElGhazaly, L ´eonie Borne, Mostafa Sadeghi, Scott Wisdom, Manuel Pariente, John Hershey, Daniel Pressnitzer, and Jon Barker. CHiME-7 UDASE evaluation data, February 2024
2024
-
[17]
The voicemos challenge 2022
Wen Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. The voicemos challenge 2022. In Interspeech 2022 , pages 4536–4540, 2022
2022
-
[18]
Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian M ¨oller. Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. In Interspeech, 2021
2021
-
[19]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
-
[20]
Data Augmentation Using Deep Generative Models for Embedding Based Speaker Recognition
Shuai Wang, Yexin Yang, Zhanghao Wu, Yanmin Qian, and Kai Yu. Data Augmentation Using Deep Generative Models for Embedding Based Speaker Recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2598–2609, 2020
2020
-
[21]
Enhancing speaker extraction through rectifying target confusion
Jiahe Wang, Shuai Wang, Junjie Li, Ke Zhang, Yanmin Qian, and Haizhou Li. Enhancing speaker extraction through rectifying target confusion. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 349–356, 2024
2024
-
[22]
Research on resnet34 improved model
Yueqing Cheng and Wanjun Yu. Research on resnet34 improved model. In 2024 9th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) , volume 9, pages 11–14. IEEE, 2024
2024
-
[23]
Fredrik Cumlin, Xinyu Liang, Victor Ungureanu, Chandan K. A. Reddy, Christian Sch ¨uldt, and Saikat Chatterjee. DNSMOS Pro: A Reduced- Size DNN for Probabilistic MOS of Speech. In Proc. Interspeech 2024 , pages 4818–4822, 2024
2024
-
[24]
The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement
Simon Leglaive, L ´eonie Borne, Efthymios Tzinis, Mostafa Sadeghi, Matthieu Fraticelli, Scott Wisdom, Manuel Pariente, Daniel Pressnitzer, and John Hershey. The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement. In 7th International Works...
2023
-
[25]
Hershey, Daniel Pressnitzer, and Jon P
Simon Leglaive, Matthieu Fraticelli, Hend ElGhazaly, L ´eonie Borne, Mostafa Sadeghi, Scott Wisdom, Manuel Pariente, John R. Hershey, Daniel Pressnitzer, and Jon P. Barker. Objective and subjective evalu- ation of speech enhancement methods in the UDASE task of the 7th CHiME c...
2025
-
[26]
Spearman rank correlation
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics , 7, 2005
2005
-
[27]
The Kendall rank correlation coefficient
Herv ´e Abdi. The Kendall rank correlation coefficient. Encyclopedia of measurement and statistics , 2:508–510, 2007
2007
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.