Universal Preference-Score-based Pairwise Speech Quality Assessment

Yang Ai; Yu-Fei Shi; Zhen-Hua Ling

REVIEW 3 major objections 5 minor 29 references

Universal Preference-Score-based Pairwise Speech Quality Assessment

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that pairwise speech quality preference can be predicted by running a shared semantic-acoustic MOS model on each sample and mapping the MOS difference through a sigmoid preference function, and that this decomposition…

desk verdict Competent, transparent empirical study of preference-from-MOS prediction; the 16-scenario grid and paired-data construction are the real contributions, but 'universal' overstates a proxy-based result. read the letter →

arxiv 2506.01455 v1 pith:OARWEWVD submitted 2025-06-02 cs.SD eess.AS

Yu-Fei Shi , Yang Ai , Zhen-Hua Ling This is my paper

classification cs.SDeess.AS

keywords speechqualityassessmentpreferencescorepairwisecomparisonmeanopinionpredictionself-supervisedrepresentationssemanticandacousticfeaturestext-to-speechvoiceconversionevaluationlisteningtest

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech quality is usually assessed one sample at a time with a mean opinion score (MOS), but comparing two speech generation systems often means deciding which of two samples sounds better. This paper proposes UPPSQA, a universal preference-score-based pairwise speech quality assessment model that turns an absolute MOS predictor into a relative preference predictor: a shared semantic-acoustic model scores the two samples, and a preference function maps their MOS difference into a preference score in $(-1,1)$. To make up for scarce preference data, the paper constructs paired training data from the BVCC MOS dataset, pairing utterances with matched content via ASR-based clustering and with unmatched content via cross-system sampling. It then evaluates preference accuracy across 16 scenarios that cross two label conditions (with and without MOS supervision), four training-testing splits, and in-domain versus out-of-domain tests. In most of those scenarios, UPPSQA beats an SSL-MOS-based baseline and a reproduced preference network, with the largest gains when absolute MOS labels are available.

What carries the argument

The load-bearing object is the preference function of Eq. (1), a sigmoid of the MOS difference that maps every pair of absolute scores into $(-1,1)$ and makes the pairwise comparison a deterministic function of the difference. It is carried by the SA-MOS model: two identical weight-sharing feature processors extract wav2vec 2.0 semantic and WavLM acoustic representations, refine each with residual connections, concatenate them, and pass the result through a BiLSTM and linear layers to produce each sample's MOS. The other key piece is the paired-data generation, which creates content-matched pairs by clustering utterances with the same ASR-recognized text and content-unmatched pairs by drawing samples from different generation systems; that construction defines the four training-testing scenarios used to test universality.

What would settle it

Take a held-out set of speech pairs with human pairwise preference judgments, including pairs whose two samples have equal or near-equal MOS but differ in prosody or speaker identity. If human choices disagree with the sign of the MOS difference on a substantial share of such pairs, the preference function cannot be a monotone function of MOS difference and the reported accuracy would not transfer to true preference. A direct comparison of model accuracy against human labels versus MOS-derived labels would reveal this gap.

Watch

Extended reading notes

Core claim

The paper's central claim is that pairwise preference between two speech samples can be recovered by decomposing the problem into two absolute MOS predictions plus a fixed preference function, rather than by training a dedicated pairwise preference network. The UPPSQA model uses a shared SA-MOS front end that combines wav2vec 2.0 semantic features with WavLM acoustic features through residual feature processors and a BiLSTM prediction head; the two predicted MOS values feed the preference function $s^{(x,y)}_p = 2/(1+e^{-(s^{(x)}_m-s^{(y)}_m)})-1$, whose sign indicates which sample is better. Training minimizes either the MOS loss plus the preference loss (when MOS labels exist) or the preference loss alone (when they do not), with the preference label derived as the sign of the MOS difference. Across the four training-testing scenarios, both label conditions, and in-domain and out-of-domain test sets, the reported accuracy is higher for UPPSQA than for the baselines in most of the 16 settings, which the authors read as evidence of universality.

Load-bearing premise

The load-bearing premise is that a listener's preference between two speech samples is fully captured by the sign of the difference of their mean opinion scores; if real preference judgments track qualities that MOS misses, then the labels and the preference function measure a proxy rather than true preference.

Editorial extensions

If this is right

Existing MOS datasets can be turned into preference training sets without new listening tests, by pairing same-content or cross-system utterances and deriving preference labels from MOS differences.
A generic absolute-quality scorer equipped with the preference function can serve as a pairwise preference predictor, so future improvements in single-sample MOS models should transfer directly to preference ranking.
Supervising with absolute MOS labels improves pairwise accuracy, especially when test content is unmatched from training; the paper reports gains above 0.1 accuracy in the m-nm scenario.
Out-of-domain preference prediction (English-trained model on Mandarin BC2019 data) remains above the baselines in most scenarios, suggesting the decomposition generalizes across languages and corpora.
When MOS labels are unavailable, training on content-unmatched pairs yields better preference accuracy than training on content-matched pairs, likely because the unmatched data is longer and more diverse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's accuracy is measured against labels derived from MOS scores, so it is accuracy at reproducing MOS differences; a direct test with human pairwise preference judgments would reveal whether this proxy matches real preference.
Because the preference function is differentiable and fixed, the same decomposition could be appended to any MOS predictor, turning data collected for absolute scoring into a pairwise ranker without architectural changes.
The ASR-clustering method for content-matched pair construction could be reused to build preference datasets in other languages and domains, reducing the cost of preference data collection.
If listener preference is non-monotone in MOS on some pairs, the fixed sigmoid-of-difference function caps the achievable accuracy; a learnable monotone transformation of the MOS difference would be a natural testable extension.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: this is a solid, clearly-written empirical paper that builds a preference-scoring model by stacking MOS prediction with a fixed sigmoid, and evaluates it across a well-designed grid of 16 train/test scenarios. The dataset construction - content-matched pairs via ASR+clustering, content-unmatched pairs via system grouping - is genuinely useful and likely the most reusable contribution. The finding that MOS supervision helps when transferring from matched to unmatched content is plausible and worth having in the literature.

The soft spots are real but not fatal. The biggest one is construct validity: every preference label is sgn(MOS difference), so the model is trained and tested on a proxy for listener preference, not on actual A/B preference judgments. The paper says this explicitly in Section 2.3, so it is not hidden, but the abstract's "universal" claim overstates what was tested. The reported ACC values measure agreement with the proxy, and all 16 scenarios inherit that assumption. A revision should soften the claim and, ideally, add a small validation set of real preference ratings.

Second, there are no significance tests or error bars. Five-seed averaging is reported, but without variance readers cannot tell whether a 0.003 gap means anything. Several table cells show ties or losses (e.g., in-domain LA nm-nm, out-of-domain LM m-nm and nm-m), yet the abstract says "outperforms" without qualification. That needs fixing.

Third, the baselines: UTP* removes the ID embedding, but the original UTP numbers are not shown, and NORESQA and UTP2 are cited but never compared. At least report the original configuration or justify the modification with numbers.

The math is simple and sound; no red flags in the architecture. The citation pattern is fine.

Who is this for: practitioners building automatic speech quality assessment for TTS/VC evaluation, and researchers working on preference prediction from MOS data. It deserves a serious referee; the experimental design is systematic and the dataset construction is a real contribution. My recommendation: send to peer review with a request for a major revision that narrows the universality claim, adds uncertainty quantification, and either validates on real preference data or explicitly frames the task as proxy preference prediction.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes UPPSQA, a universal preference-score-based pairwise speech quality assessment model. UPPSQA first predicts absolute MOS scores for the two samples in a pair with a shared SA-MOS model combining wav2vec2 and WavLM features, then converts the MOS difference into a preference score via the sigmoid preference function in Eq. (1). To address the scarcity of preference data, the authors construct pairwise training/test sets from the BVCC and BC2019 MOS datasets, using ASR-based clustering for content-matched pairs and system-pair sampling for content-unmatched pairs (Section 2.3). Experiments cover four training-testing scenarios (m-m, nm-m, m-nm, nm-nm) under two label conditions (LA and LM) and both in-domain and out-of-domain test sets, comparing accuracy with SSLSQA and UTP* baselines. The paper concludes that UPPSQA is universal and outperforms baselines.

Significance. If the central claim holds, the paper offers a practical and economical design: a MOS prediction model can be reused as a pairwise preference predictor, and the proposed paired-data construction from MOS datasets mitigates the scarcity of preference labels. The experimental matrix is broad for this area, covering content-matched and content-unmatched training/testing, two label supervision conditions, and an out-of-domain Mandarin test set. Strengths include the use of public datasets, five-seed averaging for robustness, and a systematic comparison against two baselines. However, because every preference label in the study is synthesized as the sign of MOS differences, the reported accuracies validate agreement with a proxy rather than with actual human preference judgments; this is the main gap separating the paper's claims from its evidence.

major comments (3)

[Section 2.2, Eq. (1), Section 2.3] The construct validity of the preference labels is load-bearing and not established. Every preference label in all scenarios is defined as s_p = sgn(s_m^(x) - s_m^(y)) from mean MOS scores, and the predicted preference is the sigmoid of the predicted MOS difference in Eq. (1). Therefore, ACC in Tables 1 and 2 measures how well UPPSQA predicts this MOS-derived proxy, not how well it predicts actual listener preference. If real A/B preference judgments are influenced by factors beyond mean MOS (e.g., prosody, speaker identity, or content naturalness), the model could achieve high accuracy on the synthetic labels while misranking pairs for human listeners. The paper should validate on a real preference listening test dataset (e.g., human A/B judgments collected for TTS or VC systems) or explicitly reframe the contribution as MOS-difference prediction. Without such validation, the 'preference-score-based' claim in the title and abstract is unsupported.
[Abstract, Conclusion, Tables 1 and 2] The unqualified claim that 'the prediction accuracy of UPPSQA outperforms that of the baseline models' is stronger than the reported results. Several cells show ties or losses: Table 1 (LA, nm-nm) reports UPPSQA 0.823 against SSLSQA 0.825; Table 2 (LA, nm-m) reports 0.680 against SSLSQA 0.681; Table 2 (LM, m-nm) reports 0.594 against UTP* 0.606. In addition, no error bars, confidence intervals, or significance tests accompany the five-seed averages, so differences of 0.001-0.01 in many cells may not be meaningful. The conclusion's 'in most scenarios' is accurate, but the abstract and conclusion should be adjusted to state the qualified result, or the authors should add statistical significance testing to support the stronger claim.
[Section 3.1 and Section 5] The claim of 'universality' is broader than the evidence. The out-of-domain evaluation uses only one dataset (BC2019, Mandarin) and the in-domain evaluation uses only BVCC (English), both drawn from Blizzard Challenge, Voice Conversion Challenge, and ESPNet TTS/VC systems. This does not cover other speech types (e.g., spontaneous speech, emotional speech, or generative speech from non-TTS systems) or languages beyond English and Mandarin. The term 'universal' should be either explicitly bounded to the tested conditions or supported by a wider range of domains and languages.

minor comments (5)

[Abstract] The model name is written as 'UPP-SQA' in the abstract but 'UPPSQA' elsewhere; please use a single consistent spelling.
[Eq. (4)] The summation uses index n in 'sum_{n=1}^N' but the preference scores are written with superscripts (x_i, y_i); the subscript should be n or the notation should be unified.
[Figure 3] Figure 3 would be easier to interpret with labeled axes and the reported correlation coefficient explicitly stated in the caption or text.
[Section 4.1] The statement that models trained with content-matched and content-unmatched data achieved 'similar accuracy' on the same test set under the LA condition is made without numerical support; please report the actual values or refer to the tables more precisely.
[Section 2.2] The final loss L = L_m + L_p uses equal weights without discussion; if the two loss terms have different scales, a weighting or normalization should be justified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the preference score is an explicitly defined monotone transform of MOS, so the reported accuracy is transparent pairwise MOS-difference agreement, not a hidden reintroduction of the target.

full rationale

The paper's preference target is defined operationally: Section 2.2 sets s_p = sgn(s_m^x - s_m^y), and Eq. (1) defines the predicted preference as a monotone function of the predicted MOS difference. This is an explicit modeling choice, not a concealed loop: the model learns MOS from external human MOS ratings, and the held-out test sets provide genuine generalization signal for the defined target. The reported ACC in Eq. (4) is indeed pairwise sign agreement of MOS differences under that definition, so the 'preference' claim should be read as a claim about an MOS-derived proxy rather than about independently collected A/B preference judgments; that is a construct-validity limitation, not circularity. The only author self-citation (SAMOS, ref. [5]) is background and not load-bearing. One manuscript-internal gap is that Section 2.2 describes the LM condition as 'preference data that is obtained directly, rather than derived from MOS data,' yet Section 2.3 constructs all experimental pairs from MOS labels, so the LM scenario does not actually test direct preference labels; this weakens support for the 'universal across label conditions' claim but does not create a circular derivation. No equation is fitted to its own prediction, and no parameter is renamed as a result, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No ad hoc physical constants are fitted. The two free parameters listed are a hand-chosen sigmoid scale and an unreported DBSCAN parameter; neither changes the sign-based accuracy in a fundamental way. The main assumptions are the monotone mapping from MOS to preference and the reliability of ASR-based content clustering.

free parameters (2)

Preference function steepness = The sigmoid in Eq. (1) has no temperature parameter, so the scale is effectively 1; this value was chosen by hand.
The sigmoid scale does not affect the sign of the predicted preference, so it does not change classification accuracy, but it is a hand-chosen constant.
DBSCAN epsilon and min_samples = The paper does not report the epsilon and min_samples values used for DBSCAN clustering in Section 3.1.
These parameters determine which utterances are grouped as content-matched, affecting the constructed training data. They are chosen without reported tuning.

assumptions (3)

domain assumption Preference is a monotone function of absolute MOS.
Eq. (1) and Section 2.3 define preference labels as sgn(MOS_x - MOS_y); if real preferences are not monotone in MOS, the task is misspecified.
domain assumption ASR transcriptions correctly capture content for matching.
Section 3.1 uses the xlsr-53 ASR model and DBSCAN to group utterances by text; ASR errors can put different-content clips in the same group.
domain assumption BVCC and BC2019 listener averages are reliable ground truth.
The paper averages ratings from 8 to 17 listeners per utterance but does not assess inter-rater reliability or listener bias.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Universal Preference-Score-based Pairwise Speech Quality Assessment." pith.science (2026). https://pith.science/paper/OARWEWVD

@misc{pith2026250601455,
  author       = {Pith},
  title        = {Pith review of: Universal Preference-Score-based Pairwise Speech Quality Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OARWEWVD}},
  note         = {Machine review of arXiv:2506.01455}
}

read the original abstract

To compare the performance of two speech generation systems, one of the most effective approaches is estimating the preference score between their generated speech. This paper proposes a novel universal preference-score-based pairwise speech quality assessment (UPPSQA) model, aimed at predicting the preference score between paired speech samples to determine which one has better quality. The model first predicts the absolute mean opinion score (MOS) for the two speech samples separately, and then aggregates them into a relative preference score using a preference function. To address the scarcity of preference data, we also construct a new pairwise speech dataset based on a MOS dataset for experiments. Experimental results confirm that, whether in training scenarios with different data types and label conditions, or in both in-domain and out-of-domain test scenarios, the prediction accuracy of UPP-SQA outperforms that of the baseline models, demonstrating its universality.

Figures

Figures reproduced from arXiv: 2506.01455 by the authors.

**Figure 2.** Overview of four training-testing scenarios. When sˆ (x,y) p > 0, x has a higher quality than y; when sˆ (x,y) p = 0, x and y have equivalent quality; and when sˆ (x,y) p < 0, x has a lower quality than y. 2.2. Training Criteria We design two training methods for two different data label scenarios as follows. • LA: Scenarios where MOS labels are available. In this case, we define losses on both MOS and preference s… view at source ↗

**Figure 3.** The scatter plot showing the correlation between the accuracy (ACC) and the system-level SRCC on the development set for each checkpoint during the training with contentunmatched data under the LA label condition for UPPSQA. model is trained using single speech samples, and during testing, the model respectively calculates the MOS scores for the two compared speech samples, and then the preference score is obtain… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

[1]

Participants are asked to listen to the speech samples to be evaluated and rate one or more samples based on specific criteria

Introduction Human subjective ratings are the gold standard for assessing speech generation systems such as text-to-speech (TTS) and voice conversion (VC). Participants are asked to listen to the speech samples to be evaluated and rate one or more samples based on specific criteria. One of the most common subjec- tive evaluation method is the mean opinion...

work page
[2]

four types of data combination, where four training-testing datasets are constructed based on whether the paired speech samples have matched content; 2) two types of training label data, i.e., whether MOS labels can be utilized; and 3) two test- ing methods, i.e., using both in-domain and out-of-domain data for testing. Experimental results confirm that t...

work page
[3]

Proposed Method In this section, we will first introduce the architecture of UPP- SQA, followed by the training approach and dataset construc- tion method for UPPSQA. 2.1. Architecture Figure 1 illustrates the architecture of the proposed UPPSQA. We used an identical SA-MOS model to respectively score two speech samples in a pair and then fed the two abso...

work page Pith review arXiv 2025
[4]

Comparison with Baselines Table 1 presents the in-domain experimental results across eight scenarios (i.e., two label conditions×four training-testing sce- narios)

Experimental Results and Analysis 4.1. Comparison with Baselines Table 1 presents the in-domain experimental results across eight scenarios (i.e., two label conditions×four training-testing sce- narios). In most scenarios, our proposed UPPSQA outper- formed the two baselines, confirming its universality. The UPPSQA outperformed SSLSQA, confirming the effe...

work page arXiv
[5]

Dataset We conducted experiments using two MOS datasets

Experimental Setups 3.1. Dataset We conducted experiments using two MOS datasets. The BVCC dataset [3] was used for in-domain training and testing, while the BC2019 dataset [13] was just used for out-of-domain testing. The BVCC dataset comprised 7,106 English utterances, of- ficially divided into training, development, and test sets in a ra- tio of 70%/15...

work page 2019
[6]

When is it better to compare than to score?

N. B. Shah, S. Balakrishnan, J. Bradley, A. Parekh, K. Ramchan- dran, and M. Wainwright, “When is it better to compare than to score?”arXiv preprint arXiv:1406.6618, 2014

arXiv 2014
[7]

Nevertheless, the proposed UPPSQA still maintained stable performance, with accuracy higher than the baseline in most scenarios

The accuracy on the out-of-domain test set shows a signifi- cant drop compared to the in-domain dataset, which is reason- able, as this test set is completely unseen and contains differ- ent language content. Nevertheless, the proposed UPPSQA still maintained stable performance, with accuracy higher than the baseline in most scenarios. The out-of-domain e...

work page
[8]

Conclusion This paper proposed UPPSQA, a universal preference-score- based pairwise speech quality assessment model. UPPSQA uses a semantic-acoustic-driven SA-MOS model to predict the absolute scores of paired speech samples, and then outputs the relative preference scores through a preference function. We designed a total of 16 application scenarios base...

work page

Show all 29 references

[9]

MOSNet: Deep learning-based objec- tive assessment for voice conversion,

C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y . Tsao, and H.-M. Wang, “MOSNet: Deep learning-based objec- tive assessment for voice conversion,” inProc. Interspeech, 2019, pp. 1541–1545

2019
[10]

LDNet: Unified listener dependent modeling in mos prediction for syn- thetic speech,

W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNet: Unified listener dependent modeling in mos prediction for syn- thetic speech,” inProc. ICASSP, 2022, pp. 896–900

2022
[11]

Generaliza- tion ability of mos prediction networks,

E. Cooper, W.-C. Huang, T. Toda, and J. Yamagishi, “Generaliza- tion ability of mos prediction networks,” inProc. ICASSP, 2022, pp. 8442–8446

2022
[12]

UTMOS: UTokyo-SaruLab system for V oice- MOS Challenge 2022,

T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab system for V oice- MOS Challenge 2022,” inProc. Interspeech, 2022, pp. 4521– 4525

2022
[13]

SAMOS: A neural MOS prediction model leveraging semantic representations and acoustic features,

Y .-F. Shi, Y . Ai, Y .-X. Lu, H.-P. Du, and Z.-H. Ling, “SAMOS: A neural MOS prediction model leveraging semantic representations and acoustic features,” inProc. ISCSLP, 2024, pp. 199–203

2024
[14]

SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis,

G. Maniati, A. Vioni, N. Ellinas, K. Nikitaras, K. Klapsas, J. S. Sung, G. Jho, A. Chalamandaris, and P. Tsiakoulis, “SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis,” inProc. Interspeech, 2022, pp. 2388– 2392

2022
[15]

Wav2vec 2.0, widely used in var- ious speech tasks such as automatic speech recognition (ASR), is pre-trained on hundreds to thousands of hours of speech data

and WavLM [16] to extract semantic and acoustic features from the speech samples to be evaluated, performing a compre- hensive assessment that fully leverages the multi-dimensional characteristics of the speech. Wav2vec 2.0, widely used in var- ious speech tasks such as automa...
[16]

The reliability of the itu- p. 85 standard for the evaluation of text-to-speech systems,

Y . Vazquez-Alvarez and M. Huckvale, “The reliability of the itu- p. 85 standard for the evaluation of text-to-speech systems,” in Proc. ISCSLP, 2002, pp. 329–332

2002
[17]

Best-worst scaling more reli- able than rating scales: A case study on sentiment intensity anno- tation,

S. Kiritchenko and S. Mohammad, “Best-worst scaling more reli- able than rating scales: A case study on sentiment intensity anno- tation,” inProc. ACL, 2017, pp. 465–470

2017
[18]

Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks,

C. Valentini-Botinhao, M. S. Ribeiro, O. Watts, K. Richmond, and G. E. Henter, “Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks,” inProc. Interspeech, 2022, pp. 471–475

2022
[19]

NORESQA: A framework for speech quality assessment using non-matching references,

P. Manocha, B. Xu, and A. Kumar, “NORESQA: A framework for speech quality assessment using non-matching references,” Advances in neural information processing systems, vol. 34, pp. 22 363–22 378, 2021

2021
[20]

Preference-based training framework for automatic speech quality assessment using deep neural network,

C.-H. Hu, Y . Yasuda, and T. Toda, “Preference-based training framework for automatic speech quality assessment using deep neural network,” inProc. Interspeech, 2023, pp. 546–550

2023
[21]

Embedding learning for preference-based speech quality assessment,

C. Hu, Y . Yasuda, and T. Toda, “Embedding learning for preference-based speech quality assessment,” inProc. Inter- speech, 2024, pp. 2685–2689

2024
[22]

The Blizzard Challenge 2019,

Z. Wu, Z. Xie, and S. King, “The Blizzard Challenge 2019,” in Proc. Blizzard Challenge Workshop, 2019

2019
[23]

wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

2020
[24]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

2022
[25]

Gaussian error linear units (gelus),

D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),”arXiv preprint arXiv:1606.08415, 2016

2016 arXiv
[26]

Deep sparse rectifier neural networks,

X. Glorot, A. Bordes, and Y . Bengio, “Deep sparse rectifier neural networks,” inProceedings of the fourteenth international confer- ence on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 315–323

2011
[27]

ESP- Net: End-to-end speech processing toolkit,

S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. E. Y . Soplin, J. Heymann, M. Wiesner, N. Chenet al., “ESP- Net: End-to-end speech processing toolkit,” inProc. Interspeech, 2018, pp. 2207–2211

2018
[28]

Simple and Effective Zero-shot Cross-lingual Phoneme Recognition,

Q. Xu, A. Baevski, and M. Auli, “Simple and Effective Zero-shot Cross-lingual Phoneme Recognition,” inProc. Interspeech, 2022, pp. 2113–2117

2022
[29]

A density-based algorithm for discovering clusters in large spatial databases with noise,

M. Ester, H.-P. Kriegel, J. Sander, X. Xuet al., “A density-based algorithm for discovering clusters in large spatial databases with noise,” inkdd, vol. 96, no. 34, 1996, pp. 226–231

1996

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Participants are asked to listen to the speech samples to be evaluated and rate one or more samples based on specific criteria

Introduction Human subjective ratings are the gold standard for assessing speech generation systems such as text-to-speech (TTS) and voice conversion (VC). Participants are asked to listen to the speech samples to be evaluated and rate one or more samples based on specific criteria. One of the most common subjec- tive evaluation method is the mean opinion...

work page

[2] [2]

four types of data combination, where four training-testing datasets are constructed based on whether the paired speech samples have matched content; 2) two types of training label data, i.e., whether MOS labels can be utilized; and 3) two test- ing methods, i.e., using both in-domain and out-of-domain data for testing. Experimental results confirm that t...

work page

[3] [3]

Proposed Method In this section, we will first introduce the architecture of UPP- SQA, followed by the training approach and dataset construc- tion method for UPPSQA. 2.1. Architecture Figure 1 illustrates the architecture of the proposed UPPSQA. We used an identical SA-MOS model to respectively score two speech samples in a pair and then fed the two abso...

work page Pith review arXiv 2025

[4] [4]

Comparison with Baselines Table 1 presents the in-domain experimental results across eight scenarios (i.e., two label conditions×four training-testing sce- narios)

Experimental Results and Analysis 4.1. Comparison with Baselines Table 1 presents the in-domain experimental results across eight scenarios (i.e., two label conditions×four training-testing sce- narios). In most scenarios, our proposed UPPSQA outper- formed the two baselines, confirming its universality. The UPPSQA outperformed SSLSQA, confirming the effe...

work page arXiv

[5] [5]

Dataset We conducted experiments using two MOS datasets

Experimental Setups 3.1. Dataset We conducted experiments using two MOS datasets. The BVCC dataset [3] was used for in-domain training and testing, while the BC2019 dataset [13] was just used for out-of-domain testing. The BVCC dataset comprised 7,106 English utterances, of- ficially divided into training, development, and test sets in a ra- tio of 70%/15...

work page 2019

[6] [6]

When is it better to compare than to score?

N. B. Shah, S. Balakrishnan, J. Bradley, A. Parekh, K. Ramchan- dran, and M. Wainwright, “When is it better to compare than to score?”arXiv preprint arXiv:1406.6618, 2014

arXiv 2014

[7] [7]

Nevertheless, the proposed UPPSQA still maintained stable performance, with accuracy higher than the baseline in most scenarios

The accuracy on the out-of-domain test set shows a signifi- cant drop compared to the in-domain dataset, which is reason- able, as this test set is completely unseen and contains differ- ent language content. Nevertheless, the proposed UPPSQA still maintained stable performance, with accuracy higher than the baseline in most scenarios. The out-of-domain e...

work page

[8] [8]

Conclusion This paper proposed UPPSQA, a universal preference-score- based pairwise speech quality assessment model. UPPSQA uses a semantic-acoustic-driven SA-MOS model to predict the absolute scores of paired speech samples, and then outputs the relative preference scores through a preference function. We designed a total of 16 application scenarios base...

work page