{"id":"8158ba47-b335-45b1-a2e5-c2a94621f857","arxiv_id":"2506.23874","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A pair-comparison ranking model trained on one small challenge dataset ranks speech enhancement systems better than DNSMOS and UTMOS across multiple languages and domains.","lead":"URGENT-PK is a speech-quality ranking model that compares two enhanced speech samples and aggregates the pairwise decisions to rank entire speech enhancement systems. It outperforms standard MOS predictors like DNSMOS and UTMOS on held-out challenge data, suggesting that pairwise comparison is a data-efficient alternative for system evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The data-cleaning threshold δ is selected using the same urgent25en test set that produces the headline ranking results, so the primary in-domain comparison is not fully held out; 8-utterance validation and absent error bars add uncertainty.","rationale":"The reader's weakest assumption correctly flags the fragility of an 8-utterance validation set and the MOS-difference proxy for pairwise preference. My stress-test identifies a more specific and more objective problem: the data-cleaning threshold δ is justified by an ablation on urgent25en, the same set used for the headline Table III results. This makes the primary in-domain evaluation partly circular unless δ was fixed before any look at urgent25en. I do not think this overturns the paper: the multilingual urgent25zh/jp/de and CHiME-7 results are independent of the δ choice, and the overall pattern of improvement over DNSMOS and UTMOS is consistent. However, the central claim as stated includes urgent25en, and the lack of significance testing means the reported margins could be smaller than they appear. The paper's comparative architecture and multi-dataset evaluation are real evidence, and the weaknesses are addressable with additional held-out validation and uncertainty quantification. The reader's CONDITIONAL verdict remains appropriate, so I recommend no change.","tokens_in":12084,"tokens_out":8189,"duration_ms":100644,"concrete_test":"Hold out urgent25en completely while tuning: select δ and all other hyperparameters using only urgent24en (e.g., an internal validation split or nested cross-validation), then freeze the configuration and evaluate once on urgent25en. Report results across at least 5 training seeds with mean and standard deviation, plus a bootstrap confidence interval over the 22 systems for the delta between URGENT-PK and UTMOSf t on KRCC, SRCC, and LCC. If the margin over UTMOSf t is no longer outside the confidence interval, the headline superiority on the primary English test set is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main in-domain result in Table III is not convincingly held out. Section III-B says the MOS-difference threshold δ=0.3 is justified by the ablation in Section III-F4, and that ablation is run on urgent25en — the same test set reported in Table III. If δ=0.3 was chosen by looking at urgent25en system-level correlations, the Table III margins over UTMOSf t (e.g., KRCC 0.879 vs 0.814) are partly the result of test-set tuning and may not reproduce on a fresh English test set. The multilingual and CHiME-7 results in Table II are not affected by this particular choice, so the central claim may still survive; but the paper's primary in-domain evidence is weaker than it appears. Separately, checkpoint selection uses an 8-utterance validation split (Section III-E), and no confidence intervals or significance tests are reported, so it is unclear whether the observed differences are within sampling noise, especially for the 5-system CHiME-7 set where KRCC/SRCC have coarse resolution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes URGENT-PK, a pairwise comparison model for ranking speech enhancement (SE) systems. The utterance-level model takes two homologous enhanced speech samples produced from the same noisy input, encodes them with either a log-mel front end or a UTMOS-based encoder, and uses a ResNet34 comparison module to output a comparative score plus two MOS estimates. The system-level Enumerating-Comparing-Scoring (ECS) algorithm accumulates wins or scores over all system pairs. The model is trained on the urgent24en dataset with a multi-task objective combining pairwise BCE loss and MOS MSE loss, and is evaluated on urgent25en, urgent25zh/ja/de, and CHiME-7 UDASE by computing system-level KRCC, SRCC, and LCC against oracle MOS. The authors report that URGENT-PK variants outperform DNSMOS, UTMOS, fine-tuned UTMOS, and UTMOS with ECS on most test sets, despite a simple architecture and limited training data.","tokens_in":12253,"tokens_out":4356,"duration_ms":54963,"significance":"If the empirical claims are robust, the paper makes a useful contribution: it shows that a pairwise ranking paradigm can exploit limited paired MOS data more efficiently than absolute MOS prediction, and that system-level rankings can be obtained from an interpretable voting algorithm. The multi-task design (pairwise score plus predicted MOS) is a reasonable way to inject perceptual knowledge, and the evaluation on multiple languages and a different challenge (CHiME-7 UDASE) is a genuine strength. The authors also honestly report the one clear failure case (UG-PKmel on urgent25zh) and provide an ablation of the data-cleaning threshold. However, the evaluation has load-bearing robustness issues: the main in-domain test set was used to choose a hyperparameter, the validation set has only 8 utterances, and no confidence intervals or significance tests are reported. These issues do not necessarily invalidate the central claim, but they need to be addressed before the reported superiority can be taken at face value.","major_comments":[{"comment":"The data-cleaning threshold δ=0.3 is selected by an ablation performed on urgent25en (Section III-F4, Figure 2), and the same urgent25en set is the test set for the headline results in Table III. This means the in-domain English comparison is not fully held out: the threshold was chosen by looking at exactly the systems and utterances whose correlations are then reported as the main evidence. The margins over UTMOSft (e.g., KRCC 0.879 vs. 0.814) could therefore be partially inflated by test-set tuning. Please re-select δ using the validation split or an inner cross-validation on the training split, and re-run Table III with that chosen value; if the conclusions change, the abstract's claim about 'multiple open test sets' should be rephrased to emphasize the multilingual and CHiME-7 results, which are not affected by this particular tuning step.","section":"Section III-F4 / Figure 2 / Table III"},{"comment":"The validation set used for checkpoint selection contains only 8 utterances, and no confidence intervals, bootstrap estimates, or significance tests are reported anywhere in the paper. System-level correlations on 22 systems (urgent25) and especially on 5 systems (CHiME-7) have coarse resolution: for CHiME-7, a single system reordering can change KRCC or SRCC by a large discrete step, so differences such as 0.400 vs. 0.200 or 0.500 vs. 0.300 may easily be within sampling noise. Please report confidence intervals (e.g., bootstrap over systems or over utterances) for the correlation values and, where meaningful, a paired significance test between URGENT-PK and the strongest baseline. Without this, the reader cannot tell whether the observed gaps are systematic or accidental.","section":"Section III-E / Tables II and III"},{"comment":"The loss weights α=β=0.5 and the number of mel filters (120) are presented as fixed choices without ablations or sensitivity analysis, and the data-cleaning threshold δ is the only hyperparameter studied. Since the paper's argument is that the pairwise paradigm is data-efficient and robust, it would strengthen the paper to show that the main conclusions are not sensitive to reasonable variations of α, β, and the mel filter count. At minimum, please state whether these values were chosen on the 8-utterance validation set, and if so, acknowledge the instability risk of tuning on such a small set.","section":"Section III-B and Section III-C"}],"minor_comments":[{"comment":"The axis labels and bar values in Figure 2 are rendered as garbled unicode replacement sequences (e.g., '/uni00000013/uni00000011/...'), making the actual threshold values and correlation values unreadable. Please replace the figure with a clean version and, ideally, also provide the numeric values in a table.","section":"Figure 2"},{"comment":"The row label 'MOScp' is not defined. The text describes a comparison between the pairwise model and 'MOS comparison', but it is unclear whether MOScp denotes the accuracy of comparing the oracle MOS values, the predicted MOS values, or something else. Please define the row explicitly.","section":"Table IV"},{"comment":"There is a typo in 'fundamentally differeserent' and later in Section II-B2 the text says 'In the Binary Scoring (BS) strategy' when describing the non-binary strategy; these should be corrected.","section":"Section II-A"},{"comment":"In the range notation '[0,0.4)' and similar, there are missing spaces after commas; also the phrase 'no more confusing' should likely be 'no longer confusing'. These small language and formatting issues should be fixed in revision.","section":"Section III-B"},{"comment":"The paper does not state whether the URGENT-PK training code, model checkpoints, or inference scripts will be released. Given the reproducibility-oriented evaluation on open challenge data, please add an availability statement or explain why the artifacts are not released.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the multilingual/CHiME-7 results are genuinely interesting, but the threshold-tuning issue on urgent25en is a real methodological flaw that can be fixed by rerunning with δ chosen on the validation set. I would also like to see confidence intervals, because the system-level N is small and several headline differences may not be significant. If the authors can provide both, the paper could become acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, this is a genuinely useful application of pairwise ranking to speech enhancement evaluation, and the cross-language/domain generalization is the real result. Second, the paper's strongest table (urgent25en, Table III) is not as clean as it looks: the data-cleaning threshold δ=0.3 is selected by running an ablation on urgent25en itself (Section III-F4, Figure 2), so part of the margin over UTMOS on that same set is test-set tuning. The multilingual and CHiME-7 tables are not affected by that choice, so the central claim — a pairwise model trained on one small dataset ranks unseen SE systems better than standard MOS predictors — still has real support, but the English headliner should be treated with caution.\n\nWhat's new: an explicit comparison module in the architecture (rather than just a pairwise loss like RankNet/MicRank) and an ECS aggregation that enumerates system pairs and accumulates wins/scores. That's incremental but sensible, and the ablations show both pieces matter. The evaluation is mostly well done: held-out multilingual test sets, CHiME-7, comparison against DNSMOS/UTMOS and fine-tuned UTMOS, and a subjective A/B sanity check. The data-cleaning idea is reasonable, and the threshold ablation (on the wrong set) at least shows the model is not hypersensitive in the 0.2–0.4 range.\n\nSoft spots, in proportion. The 8-utterance validation set for checkpoint selection is extremely small; with no error bars or significance tests, we don't know if differences like KRCC 0.853 vs 0.835 are real. The CHiME-7 set has only 5 systems, so the 0.4 vs 0.2 KRCC differences are coarse — the paper admits this. The 'perceptually-aligned' framing overstates what MOS-derived binary labels can support; the comparison signal is one step removed from actual human preferences. None of these kill the paper, but all are fixable and should be fixed.\n\nBottom line: this deserves a serious referee. It's an incremental but solid contribution to SE evaluation methodology, and the multilingual generalization is worth having. I'd suggest the editor ask for the δ selection moved to validation-only, CIs or bootstrap estimates, and a clear statement about the 8-utterance validation. I'd bring it to reading group and would cite it if I worked on SE scoring.\n\nFor peer review: accept the paper into the review process, not desk reject.","headline":"A useful pairwise ranking model for speech enhancement evaluation with real cross-language evidence, but the headline English result is partly compromised by test-set tuning of the data-cleaning threshold; the multilingual results are cleaner.","tokens_in":12863,"tokens_out":1969,"would_cite":true,"duration_ms":21257,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained on 292 English utterances, the pairwise-comparison model URGENT-PK ranks unseen speech-enhancement systems more accurately than DNSMOS and UTMOS across four languages and an unseen acoustic domain.","keywords":["speech evaluation","scoring model","Mean Opinion Score","comparison-based method","pairwise comparison","ranking","speech enhancement","system-level ranking"],"falsifier":"Re-run the full training and checkpoint-selection pipeline on several different random 8-utterance validation splits drawn from the 292 urgent24 English training utterances, and measure the spread of test-set correlations on urgent25 English; if the best URGENT-PK variant's Kendall rank correlation varies by more than roughly 0.05 across splits, the reported margins over DNSMOS and UTMOS are an artifact of the single validation split rather than a stable property of the method.","tokens_in":11841,"feed_emoji":"🎧","tokens_out":10544,"duration_ms":96895,"temperature":0.7,"pith_summary":"The paper aims to establish that ranking speech enhancement (SE) systems does not require accurate absolute quality scores: a neural model that directly compares two enhanced versions of the same noisy utterance can rank whole systems better than standard MOS predictors. The proposed URGENT-PK combines an utterance-level pairwise model with an Enumerating-Comparing-Scoring algorithm that accumulates pairwise comparisons into a system ranking. Trained on just 292 English utterances from the URGENT 2024 challenge, it reports higher system-level Kendall, Spearman, and linear correlations than DNSMOS, UTMOS, and fine-tuned UTMOS on English, Chinese, Japanese, German, and CHiME-7 UDASE test sets. The implication, if correct, is that lightweight comparison models trained on very little data can outperform MOS predictors in ranking tasks and transfer across languages and acoustic domains.","feed_headline":"Pairwise model beats MOS predictors at ranking speech systems","feed_subtitle":"Trained on 292 English utterances, URGENT-PK ranks unseen systems in four languages better than DNSMOS and UTMOS.","key_machinery":"The load-bearing machinery is the pairing of an utterance-level pairwise model with a system-level Enumerating-Comparing-Scoring (ECS) algorithm. The pairwise model takes two enhanced versions of the same noisy utterance, encodes them separately through either a log-mel spectrogram encoder or a UTMOS-based encoder (frozen or fine-tuned), concatenates the temporal embeddings along a new channel dimension, and feeds the fused representation to a modified ResNet34. The network outputs a comparative score in $[0,1]$ and estimated MOS values for both inputs. The ECS algorithm then enumerates all $K(K-1)/2$ system pairs, runs the pairwise model over all $M$ utterances for each pair, and accumulates one point to the winner (binary scoring) or fractional scores proportional to the comparative score (non-binary scoring), with the final ranking given by the accumulated scores. The paper emphasizes the explicit comparison module as the architectural departure from RankNet and MicRank, which encode ranking relationships only through their loss functions.","core_discovery":"URGENT-PK's central discovery is that a pairwise comparison paradigm extracts far more supervision from scarce MOS-labeled data than direct MOS regression, because every pair of systems processing the same noisy input yields a comparison and the $K(K-1)/2$ system pairs multiply the effective training signal. The utterance-level model encodes two homologous speech samples (the same noisy utterance enhanced by two different systems), concatenates the embeddings, and passes them through a ResNet34-based comparison module that outputs a comparative score in $[0,1]$ along with estimated MOS for both inputs; training minimizes a binary cross-entropy loss on the comparison against the sign of the MOS difference plus an MSE loss on the MOS estimates. At inference, the ECS algorithm enumerates all system pairs, scores every utterance pair, and accumulates either binary or fractional points per system to produce the final ranking. On urgent25 English, the best URGENT-PK variant (with fine-tuned UTMOS encoder) reaches a Kendall rank correlation of 0.879 and a Spearman correlation of 0.972, and even the log-mel variant trained from scratch outperforms fine-tuned UTMOS, a pattern the authors say holds on multilingual and cross-domain test sets. They interpret these results as evidence that a simple, data-efficient comparison model can act as a perceptually aligned ranking engine for SE competitions.","pith_inferences":["Because the labels are derived from averaged MOS differences rather than direct pairwise human judgments, the model inherits any bias in the original listening test; collecting true A/B preference labels on the same utterances might yield a cleaner training signal and likely shift the optimal MOS-difference threshold.","The ECS aggregation treats every utterance pair equally; weighting comparisons by confidence or by the size of the comparative margin could improve ranking stability when systems are close in quality.","The pairwise model only compares systems on the same noisy input, so it cannot assess cross-condition consistency; a test that mixes noisy conditions within a pair would probe whether the learned comparison transfers to conditions not seen in training.","A natural extension is to apply the same pairwise-comparison-plus-aggregation scheme to other quality dimensions such as intelligibility, naturalness, or speaker similarity, where absolute scales are equally noisy and relative preferences may be more consistent."],"forward_implications":["Speech enhancement challenges could replace or pre-filter full MOS listening tests with a pairwise comparison model, since a few hundred labeled utterances suffice to train a competitive ranker.","The pairwise objective multiplies scarce data: with 22 systems and 150 utterances per system, each utterance yields 231 system-pair comparisons, so the effective training-set size is far larger than the raw utterance count.","The same model can also act as a MOS predictor via replication or noisy-speech prompting, giving it a fallback when absolute scores are required.","Because the log-mel variant trained from scratch is competitive, pre-trained self-supervised encoders are not necessary for good ranking performance, lowering the computational and data barrier to deployment.","The reported transfer across Chinese, Japanese, and German suggests that relative quality judgments generalize across languages better than absolute MOS scales, which could help multilingual speech quality assessment."],"supporting_citations":[{"why":"Supplies the urgent24 en training set (292 utterances for training, 8 for validation) on which all URGENT-PK variants and fine-tuned baselines are trained.","marker":"[14]"},{"why":"Provides the urgent25 English and multilingual test sets used to evaluate the model's system-level ranking against oracle MOS.","marker":"[15]"},{"why":"Provides the CHiME-7 UDASE evaluation data, the out-of-domain test set with only 5 systems used to probe cross-domain generalization.","marker":"[16]"},{"why":"DNSMOS is the first baseline MOS predictor that URGENT-PK is compared against and outperforms.","marker":"[1]"},{"why":"UTMOS is the second baseline MOS predictor, and its frozen or fine-tuned encoder is also used as one of URGENT-PK's speech encoders.","marker":"[3]"},{"why":"MicRank supplies the prior pairwise-learning formulation and the data cleaning scheme (MOS-difference threshold) that URGENT-PK adapts.","marker":"[10]"},{"why":"RankNet is the learning-to-rank baseline that URGENT-PK is contrasted with, since its comparison module makes ranking explicit in the architecture.","marker":"[13]"}],"fun_headline_variants":["Pairwise ranking beats MOS predictors on scarce data","Data-lean pairwise model tops speech ranking baselines","URGENT-PK: Pairwise comparison wins speech ranking","Speech ranking: pairwise beats MOS even with 292 clips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training labels and model selection are trustworthy: pairwise labels are created by thresholding the sign of an 8-listener average MOS difference at 0.3, and every hyperparameter and the final checkpoint are chosen using a validation set of only 8 utterances, so the claimed cross-domain generalization depends on that tiny split being representative and on the MOS-difference proxy faithfully matching human pairwise preference.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise ranking beats MOS predictors on scarce data","Data-lean pairwise model tops speech ranking baselines","URGENT-PK: Pairwise comparison wins speech ranking","Speech ranking: pairwise beats MOS even with 292 clips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3115,"prompt_tokens":968,"completion_tokens":2147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2082}},"tokens_in":584,"tokens_out":2147,"duration_ms":15934,"temperature":1.0,"reasoning_tokens":2082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:29:29.271368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full training and checkpoint-selection pipeline on several different random 8-utterance validation splits drawn from the 292 urgent24 English training utterances, and measure the spread of test-set correlations on urgent25 English; if the best URGENT-PK variant's Kendall rank correlation varies by more than roughly 0.05 across splits, the reported margins over DNSMOS and UTMOS are an artifact of the single validation split rather than a stable property of the method.","supporting_citations":[{"cited_title":"URGENT challenge: Universality, robustness, and generalizability for speech enhancement","cited_arxiv_id":null,"evidence_quote":"Supplies the urgent24 en training set (292 utterances for training, 8 for validation) on which all URGENT-PK variants and fine-tuned baselines are trained."},{"cited_title":"Interspeech 2025 URGENT speech enhancement challenge","cited_arxiv_id":null,"evidence_quote":"Provides the urgent25 English and multilingual test sets used to evaluate the model's system-level ranking against oracle MOS."},{"cited_title":"CHiME-7 UDASE evaluation data, February 2024","cited_arxiv_id":null,"evidence_quote":"Provides the CHiME-7 UDASE evaluation data, the out-of-domain test set with only 5 systems used to probe cross-domain generalization."},{"cited_title":"DNSMOS: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors","cited_arxiv_id":null,"evidence_quote":"DNSMOS is the first baseline MOS predictor that URGENT-PK is compared against and outperforms."},{"cited_title":"UTMOS: UTokyo-SaruLab system for V oiceMOS challenge 2022","cited_arxiv_id":null,"evidence_quote":"UTMOS is the second baseline MOS predictor, and its frozen or fine-tuned encoder is also used as one of URGENT-PK's speech encoders."},{"cited_title":"Learning to rank microphones for distant speech recognition","cited_arxiv_id":null,"evidence_quote":"MicRank supplies the prior pairwise-learning formulation and the data cleaning scheme (MOS-difference threshold) that URGENT-PK adapts."},{"cited_title":"Learning to rank for information retrieval","cited_arxiv_id":null,"evidence_quote":"RankNet is the learning-to-rank baseline that URGENT-PK is contrasted with, since its comparison module makes ranking explicit in the architecture."}],"review_version":1}