Pith. sign in

REVIEW 3 major objections 6 minor 27 references

SALF-MOS: Speaker Agnostic Latent Features Downsampled for MOS Prediction

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read SALF-MOS, a 1,574-parameter model built from downsampled wav2vec latent features, reports state-of-the-art MOS prediction on BVCC, VCC2018, SOMOS, and TMHINTQI without fine-tuning or pre-training any SSL model.

desk verdict SALF-MOS's SOTA claim rests on an evaluation split that doesn't match the baselines, so the result is unverified; worth a careful look only as a cautionary protocol study. read the letter →

arxiv 2506.02082 v1 pith:UBKCY3ED submitted 2025-06-02 cs.SD cs.AIcs.LG

classification cs.SDcs.AIcs.LG
keywords meanopinionscoreMOSpredictionspeechqualityassessmentwav2vecfeaturesself-supervisedlearningconvolutionalneuralnetworktext-to-speechevaluationvoiceconversion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech quality for text-to-speech and voice conversion systems is usually judged by human listeners on a five-point Mean Opinion Score, a slow and costly process. The paper tries to show that a very small neural net can replace this human rating: it predicts MOS from downsampled wav2vec representations using a U-Net-style stack of double convolutions with only 1,574 parameters. Across four open datasets, it reports the best scores among the compared systems on MSE, linear concordance correlation, Spearman rank correlation, and Kendall's tau, without fine-tuning or pre-training the self-supervised model. If this holds, cheap and fast automatic quality scoring would become practical for model selection in TTS and voice conversion development.

What carries the argument

The machinery is a U-Net-inspired downsampling tower applied to frozen wav2vec features. Four double-convolution blocks, each a 1-D convolution with kernel 3, stride 1, padding 1, followed by batch normalization and ReLU repeated twice, extract features, while three downsampling layers halve the length; after each block a linear 'latent feature extraction' layer projects the feature map, and all these projected maps are stacked and passed to a final linear layer that regresses the MOS score. The key operation is the stacking of multi-resolution latent features, which lets the network see both fine-grained and coarse temporal structure while keeping the parameter count at 1,574.

What would settle it

Run the same 1,574-parameter model on the official VoiceMOS BVCC test split (or any published fixed split) and compare LCC and SRCC with the published UTMOS, DDOS, and NORESQA-MOS numbers; if the gap disappears or reverses, the reported superiority is an artifact of the random split.

Watch

Extended reading notes

Core claim

The central claim is that speaker-agnostic latent features, obtained by taking frozen wav2vec embeddings and compressing them through three downsampling stages with four double-convolution blocks, are sufficient for accurate MOS prediction. Each double convolution runs a 1-D convolution (kernel 3, stride 1, padding 1), batch normalization, and ReLU twice; the intermediate feature maps are projected through linear layers, stacked, and mapped by a final linear layer to a single MOS score. The paper reports that this architecture with 1,574 parameters achieves, for example, LCC 0.948 and SRCC 0.946 on BVCC, and outperforms the compared baselines on MSE, LCC, SRCC, and KTAU across BVCC, VCC2018, SOMOS, and TMHINTQI. The model does not use listener IDs, domain IDs, or any fine-tuning of the SSL backbone.

Load-bearing premise

The reported state-of-the-art comparison assumes that the random 8:1:1 train/validation/test split of each dataset is equivalent to the official challenge splits used by the baselines, with no speaker or utterance appearing in both training and test.

Editorial extensions

If this is right

  • Only 1,574 parameters are needed for the regression head, so MOS scoring can run on modest hardware without storing a large fine-tuned model.
  • Since the frozen wav2vec extractor is never fine-tuned, the same backbone can be reused across datasets and languages without per-dataset retraining of the SSL model.
  • The reported ablations indicate that feature choice matters more than head size: replacing wav2vec with MFCC, LFCC, or x-vectors drops BVCC LCC from 0.948 to between 0.47 and 0.68.
  • If the results hold under the stated split, the method gives a practical tool for screening TTS and voice conversion systems before spending money on human listening tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether other frozen SSL encoders (HuBERT, WavLM) feed the same tower equally well; if the mechanism is generic latent compression, the head should transfer across SSL feature spaces.
  • The same stacked downsampled-latent design could be redirected to other perceptual ratings, such as naturalness, intelligibility, or emotion valence, by swapping only the output layer.
  • The 1,574-parameter count refers to the regressor head; deployment still requires computing wav2vec embeddings, so the end-to-end resource footprint is larger than the headline parameter count suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents SALF-MOS, a 1,574-parameter model that predicts MOS from downsampled wav2vec features using four stacked double-convolution blocks and three downsampling stages, followed by latent feature stacking and a final linear layer. The authors compare SALF-MOS against several prior MOS prediction systems on BVCC, VCC2018, SOMOS, and TMHINTQI, reporting MSE, LCC, SRCC, and KTAU, and claim state-of-the-art performance on all four datasets. The experiments use a random 8:1:1 train/validation/test split (Section III), no reported random seeds, and no released code or data. The paper also includes a feature ablation (Table III) and a model-depth study on BVCC (Fig. 5).

Significance. If the reported numbers were obtained under the same evaluation protocol as the baselines, a 1,574-parameter MOS predictor with LCC 0.948 and SRCC 0.946 on BVCC would be a notable practical contribution, as it would suggest that large SSL ensembles and fine-tuning are unnecessary for state-of-the-art MOS prediction. The paper's strengths include its very small model size, a feature ablation that tests MFCC, LFCC, x-vectors, and wav2vec representations, and evaluation across four public datasets. However, the central empirical claim is currently not verifiable because the evaluation protocol appears incomparable to that of the baselines, because the metric definitions in Section III-C contain mathematical errors, and because no seeds, error bars, code, or data are provided.

major comments (3)
  1. [Section III, Table I] The comparison in Table I is not apples-to-apples. The paper states that training, validation, and testing data are split in the ratio 8:1:1 (Section III, first paragraph), which describes a random utterance-level split, while the baseline scores cited for UTMOS, DDOS, NORESQA-MOS, Fusion-SSL, and MOSPC come from the official VoiceMOS 2022 evaluation, whose BVCC test set is system-disjoint. Since BVCC, VCC2018, SOMOS, and TMHINTQI contain multiple utterances per speaker and per synthesis system, an utterance-level random split places parts of the same speaker or system in both the training and test sets. A 1,574-parameter head on wav2vec features can then exploit speaker or recording identity rather than perceived quality, which would inflate the reported LCC 0.948 and SRCC 0.946 on BVCC. To support the SOTA claim, the authors must evaluate on the exact official splits used by each baseline, or at minimum use system-disjoint splits, and report multiple seeds with error bars.
  2. [Section III-C, Eqs. (2)-(4)] The metric equations are mathematically incorrect. Equation (2), labelled LCC, is the Pearson product-moment correlation coefficient; Lin's concordance correlation coefficient includes additional terms that account for the mean offset and the scaling difference between the two variables. Equation (3), labelled SRCC, uses sum of x_i squared minus y_i squared in the numerator, which is not the squared difference of ranks; the correct expression is 1 - 6*sum(d_i^2)/(n(n^2-1)) with d_i equal to the difference between the paired ranks. Equation (4) omits any treatment of tied ranks, which is important for MOS data with discrete 1-5 scores. Because Table I consists entirely of numbers derived from these metrics, the paper must provide the exact formulas actually used in the implementation and, ideally, release code so that the reported values can be reproduced.
  3. [Section III-E, Fig. 5] The model selection procedure biases the reported test numbers. The architecture depth (four double-convolution layers) and the input feature set were selected based on performance on BVCC (Fig. 5 and Table III), and the same BVCC test split is then used in the headline comparison. No validation set independent of the reported test set is described, no random seeds are given, and no error bars or significance tests are reported. As a result, the reader cannot determine whether the 0.948/0.946 values on BVCC are stable or within the noise of a 1,574-parameter model. The authors should either use a separate held-out set for model selection or report results across multiple seeds, and they should disclose the standard deviation of each metric.
minor comments (6)
  1. [Section II] The text contains repeated words and typos, including 'and and' in the paragraph on objective metrics and 'V oice' in the Introduction; these should be corrected.
  2. [Table I] The MBNet row reports the values '0564' and '0426' without decimal points; they should presumably be 0.564 and 0.426, respectively.
  3. [Table I and Section III-A] The dataset name is inconsistent: Table I uses 'THMINTQI' while the text and Table II use 'TMHINTQI'; please standardize throughout.
  4. [Section III-C, Eq. (1)] Equation (1) omits the summation bounds and an index on n, so the mean squared error expression is not fully defined as printed.
  5. [Table III and Table I] The KTAU value for wav2vec on BVCC is reported as 0.81 in Table III but as 0.819 in Table I for the same configuration; this discrepancy should be resolved.
  6. [Title and Section I] The term 'speaker agnostic' is not supported by any evidence in the paper. Merely omitting speaker IDs as explicit inputs does not make the model speaker-agnostic, because wav2vec features are known to encode speaker identity, and the UMAP visualization in Fig. 3 actually shows distinct clusters. A speaker-disjoint evaluation would be needed to substantiate this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SALF-MOS is a standard supervised fit evaluated on a held-out test fold; the split/protocol and metric concerns are correctness risks, not circular reasoning.

full rationale

The paper's derivation chain is a normal supervised learning pipeline: frozen wav2vec features are fed into a small convolutional/linear head trained with L1 loss on a training fold, and results are reported on a held-out test fold (Section III: "Training, Validation and Testing data are splitted int the ratio of 8:1:1 and all results shown in this paper are for testing data."). No fitted parameter is renamed as a prediction, no result is defined in terms of itself, and no load-bearing self-citation can be identified: the authors are not the authors of the cited SSL models, datasets, or baseline systems. The choice of wav2vec features and depth four was guided by validation experiments, but selecting hyperparameters/features on a validation split and then reporting test performance is standard model selection, not circularity. The strongest concerns in this paper are correctness issues rather than circularity: the random 8:1:1 split may not match the official VoiceMOS challenge partitions used by UTMOS, DDOS, MOSPC, and Fusion-SSL, and Equations (2) and (3) in Section III-C are not the standard LCC and SRCC formulas. Those issues undermine the SOTA comparison, but they do not make the derivation equivalent to its inputs by construction. The paper is therefore best scored 0 for circularity while remaining open to review on evaluation protocol and metric correctness.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's contributions rely on a small number of hand-picked hyperparameters and two untested domain assumptions. It introduces no new theoretical entities.

free parameters (4)
  • model depth = 4
    Selected based on BVCC ablation (Fig. 5) and then used for all datasets, without accounting for selection bias.
  • input feature set = wav2vec
    Chosen after comparing MFCC, LFCC, X-vector, and wav2vec on BVCC (Table III); other features performed worse.
  • learning rate = 1e-4
    Reported in Section III without tuning details; assumed chosen by hand.
  • batch size = 4
    Chosen by hand.
assumptions (3)
  • domain assumption wav2vec features are sufficient to predict MOS.
    The model relies entirely on frozen wav2vec features; no analysis of feature sufficiency is provided.
  • domain assumption An 8:1:1 random split yields a test set comparable to the baselines' evaluations.
    The paper does not specify if this matches the official VoiceMOS challenge splits used by prior work.
  • ad hoc to paper Speaker agnosticism follows from not using speaker IDs.
    The claim of being speaker agnostic is not demonstrated; UMAP plots are offered as evidence but are not a statistical test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SALF-MOS: Speaker Agnostic Latent Features Downsampled for MOS Prediction." pith.science (2026). https://pith.science/paper/UBKCY3ED

@misc{pith2026250602082,
  author       = {Pith},
  title        = {Pith review of: SALF-MOS: Speaker Agnostic Latent Features Downsampled for MOS Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBKCY3ED}},
  note         = {Machine review of arXiv:2506.02082}
}
read the original abstract

Speech quality assessment is a critical process in selecting text-to-speech synthesis (TTS) or voice conversion models. Evaluation of voice synthesis can be done using objective metrics or subjective metrics. Although there are many objective metrics like the Perceptual Evaluation of Speech Quality (PESQ), Perceptual Objective Listening Quality Assessment (POLQA) or Short-Time Objective Intelligibility (STOI) but none of them is feasible in selecting the best model. On the other hand subjective metric like Mean Opinion Score is highly reliable but it requires a lot of manual efforts and are time-consuming. To counter the issues in MOS Evaluation, we have developed a novel model, Speaker Agnostic Latent Features (SALF)-Mean Opinion Score (MOS) which is a small-sized, end-to-end, highly generalized and scalable model for predicting MOS score on a scale of 5. We use the sequences of convolutions and stack them to get the latent features of the audio samples to get the best state-of-the-art results based on mean squared error (MSE), Linear Concordance Correlation coefficient (LCC), Spearman Rank Correlation Coefficient (SRCC) and Kendall Rank Correlation Coefficient (KTAU).

Figures

Figures reproduced from arXiv: 2506.02082 by the authors.

Figure 1
Figure 1. SALF-MOS Model Architecture In this paper, we will present Speaker Agnostic Latent Features Downsampled for MOS Prediction (SALF-MOS) which is shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Data Distribution for various data sets [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. From Fig. 3 it can be observed that even though the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Box Plot for MOS on different data sets TABLE III FEATURE EXPERIMENTATION ON BVCC Feature MSE LCC SRCC KTAU MFCC 0.56 0.54 0.55 0.49 LFCC 0.43 0.683 0.68 0.62 X-Vector 0.48 0.47 0.48 0.43 wav2vec 0.144 0.948 0.946 0.81 SRCC and KTAU should be as high as possible. As th…
Figure 5
Figure 5. Figure 5: Model Depth effect on BVCC IV. CONCLUSION In this paper, we are able to show that our novel model SALF-MOS generalizes well for MOS prediction tasks and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 14 canonical work pages

  1. [1]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,”arXiv preprint arXiv:1904.05862, 2019

  2. [2]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

  3. [3]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

  4. [4]

    TERA: Self-supervised learning of transformer encoder representation for speech,

    A. T. Liu, S.-W. Li, and H.-y. Lee, “TERA: Self-supervised learning of transformer encoder representation for speech,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021

  5. [5]

    AUTOMOS: Learning a non-intrusive assessor of naturalness-of-speech,

    B. Patton, Y . Agiomyrgiannakis, M. Terry, K. Wilson, R. A. Saurous, and D. Sculley, “AUTOMOS: Learning a non-intrusive assessor of naturalness-of-speech,”arXiv preprint arXiv:1611.09207, 2016

  6. [6]

    Quality-net: An end- to-end non-intrusive speech quality assessment model based on blstm,

    S.-W. Fu, Y . Tsao, H.-T. Hwang, and H.-M. Wang, “Quality-net: An end- to-end non-intrusive speech quality assessment model based on blstm,” arXiv preprint arXiv:1808.05344, 2018

  7. [7]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” ininternational confer- ence on acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), vol. 2, 2001, pp. 749–752

  8. [8]

    Evaluation of speech representations for mos prediction,

    F. S. Oliveira, E. Casanova, A. C. Junior, L. RS Gris, A. S. Soares, and A. R. Galv ˜ao Filho, “Evaluation of speech representations for mos prediction,” inInternational Conference on Text, Speech, and Dialogue. Springer, 2023, pp. 270–282

Show all 27 references
  1. [9]

    Deep learning-based non-intrusive multi-objective speech assessment model with cross-domain features,

    R. E. Zezario, S.-W. Fu, F. Chen, C.-S. Fuh, H.-M. Wang, and Y . Tsao, “Deep learning-based non-intrusive multi-objective speech assessment model with cross-domain features,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 54–70, 2022

  2. [10]

    MOSNET: Deep learning based objective assessment for voice conversion,

    C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y . Tsao, and H.-M. Wang, “MOSNET: Deep learning based objective assessment for voice conversion,”arXiv preprint arXiv:1904.08352, 2019

  3. [11]

    MBNET: Mos prediction for synthesized speech with mean-bias network,

    Y . Leng, X. Tan, S. Zhao, F. Soong, X.-Y . Li, and T. Qin, “MBNET: Mos prediction for synthesized speech with mean-bias network,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 391–395

  4. [12]

    LDNET: Unified listener dependent modeling in mos prediction for synthetic speech,

    W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNET: Unified listener dependent modeling in mos prediction for synthetic speech,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 896–900

  5. [13]

    DDOS: A mos prediction framework utilizing domain adaptive pre-training and distribution of opinion scores,

    W.-C. Tseng, W.-T. Kao, and H.-y. Lee, “DDOS: A mos prediction framework utilizing domain adaptive pre-training and distribution of opinion scores,”arXiv preprint arXiv:2204.03219, 2022

  6. [14]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12 449– 12 460, 2020

  7. [15]

    UTMOS: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: Utokyo-sarulab system for voicemos challenge 2022,”arXiv preprint arXiv:2204.02152, 2022

  8. [16]

    Fusion of self-supervised learned models for mos prediction,

    Z. Yang, W. Zhou, C. Chu, S. Li, R. Dabre, R. Rubino, and Y . Zhao, “Fusion of self-supervised learned models for mos prediction,”arXiv preprint arXiv:2204.04855, 2022

  9. [17]

    MOSPC: Mos predic- tion based on pairwise comparison,

    K. Wang, Y . Zhao, Q. Dong, T. Ko, and M. Wang, “MOSPC: Mos predic- tion based on pairwise comparison,”arXiv preprint arXiv:2306.10493, 2023

  10. [18]

    Speech quality assessment through mos using non-matching references,

    P. Manocha and A. Kumar, “Speech quality assessment through mos using non-matching references,”arXiv preprint arXiv:2206.12285, 2022

  11. [19]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. ...

  12. [20]

    Perceptual objective listening quality assess- ment (POLQA), the third generation itu-t standard for end-to-end speech quality measurement part i—temporal alignment,

    J. G. Beerends, C. Schmidmer, J. Berger, M. Obermann, R. Ullmann, J. Pomy, and M. Keyhl, “Perceptual objective listening quality assess- ment (POLQA), the third generation itu-t standard for end-to-end speech quality measurement part i—temporal alignment,”journal of the audio ...

  13. [21]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” inIEEE international conference on acoustics, speech and signal processing (ICASSP), 2010, pp. 4214–4217

  14. [22]

    The voicemos challenge 2022,

    W.-C. Huang, E. Cooper, Y . Tsao, H.-M. Wang, T. Toda, and J. Yamag- ishi, “The voicemos challenge 2022,”arXiv preprint arXiv:2203.11389, 2022

  15. [23]

    The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,

    J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z. Ling, “The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,”arXiv preprint arXiv:1804.04262, 2018

  16. [24]

    SOMOS: The samsung open mos dataset for the evaluation of neural text-to-speech synthesis,

    G. Maniati, A. Vioni, N. Ellinas, K. Nikitaras, K. Klapsas, J. S. Sung, G. Jho, A. Chalamandaris, and P. Tsiakoulis, “SOMOS: The samsung open mos dataset for the evaluation of neural text-to-speech synthesis,” arXiv preprint arXiv:2204.03040, 2022

  17. [25]

    INQSS: a speech intelligibility and quality assessment model using a multi-task learning network,

    Y .-W. Chen and Y . Tsao, “INQSS: a speech intelligibility and quality assessment model using a multi-task learning network,”arXiv preprint arXiv:2111.02585, 2021

  18. [26]

    LE-SSL-MOS: Self-supervised learning mos prediction with listener enhancement,

    Z. Qi, X. Hu, W. Zhou, S. Li, H. Wu, J. Lu, and X. Xu, “LE-SSL-MOS: Self-supervised learning mos prediction with listener enhancement,” inIEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–6

  19. [27]

    X-vectors: Robust dnn embeddings for speaker recognition,

    D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” inIEEE international conference on acoustics, speech and signal processing (ICASSP), 2018, pp. 5329–5333

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.