Pith. sign in

REVIEW 3 major objections 4 minor 34 references

A Dataset for Automatic Assessment of TTS Quality in Spanish

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new Spanish corpus of 4,326 rated TTS clips lets automated models predict naturalness within 0.8 points on a five-point MOS scale, matching English-language results.

desk verdict A genuinely useful public Spanish MOS dataset for TTS, but per-sample label reliability is the soft underbelly and the paper overpromises on confidence intervals. read the letter →

arxiv 2507.01805 v1 pith:YN5DWTKE submitted 2025-07-02 cs.SD eess.AS

Alejandro Sosa Welford , Leonardo Pepino This is my paper
classification cs.SDeess.AS
keywords text-to-speechSpanishnaturalnessassessmentMeanOpinionScoreself-supervisedspeechmodelswav2vec2.0NISQAqualityprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the first Spanish-language dataset for automatic assessment of text-to-speech naturalness: 4,326 audio clips from 12 TTS systems and 6 human voices, rated by 92 participants on the five-point Mean Opinion Scale using a test based on ITU-T Rec. P.807. The authors validate the dataset by training two kinds of quality predictors: a fine-tuned version of an existing English-trained model, and small downstream networks built on frozen self-supervised speech representations. Both approaches reach a mean absolute error of about 0.8 on the MOS scale, comparable to previously reported English results. The authors argue that this resource fills a gap that has blocked Spanish-specific TTS evaluation and cross-linguistic generalization studies.

What carries the argument

The engine is the dataset itself: 4,326 clips spanning 12 text-to-speech systems and 6 real human voices, covering proprietary, parametric, concatenative, and neural synthesis techniques across several Spanish dialects, with vocal-tract length perturbation and phase alteration used to widen the quality range. Labels come from a listening test designed around ITU-T Rec. P.807, completed by 92 participants with response-time checks and rejection of implausible ratings. The two predictor architectures are: NISQA, a CNN-LSTM model pretrained on an English corpus and then fine-tuned on the new labels, and DenseMOS, which takes wav2vec 2.0 layer activations, averages them over time, combines the 13 layers with learned positive weights, and passes the result through a small multilayer perceptron.

What would settle it

Have a fresh panel of at least three listeners rate a few hundred of the same clips independently, compute intraclass correlation on the clip level, and retrain the models on labels averaged across raters; if per-clip agreement is far below the system-grouped ICC of 0.68 or the retrained MAE exceeds 0.8, the original labels are too noisy to support the paper's accuracy claims.

Watch

Extended reading notes

Core claim

The central discovery is that a Spanish-only corpus of modest size, with most clips rated by a single listener, is sufficient to train naturalness predictors whose error matches established English benchmarks. Fine-tuning the NISQA model on the new data reduces its mean absolute error from 0.99 to 0.81 and raises its correlation with human scores from 0.71 to 0.73, while a much smaller network trained on wav2vec 2.0 embeddings achieves a mean absolute error of 0.80 with a correlation of 0.62. The learned weights in the wav2vec-based models place the largest emphasis on the CNN encoder output rather than the deeper transformer layers, suggesting that naturalness judgments rely primarily on local acoustic detail. The paper also shows that the collected ratings separate TTS systems into statistically distinct quality tiers and that human voices receive the highest scores, with systematic variation across voices within the same system.

Load-bearing premise

The load-bearing premise is that the human ratings, most of which are single opinions per clip, are stable enough to serve as training labels; inter-rater agreement is only measured after grouping clips by system or speaker and imputing missing values, not on individual clips.

Editorial extensions

If this is right

  • Spanish TTS evaluation no longer has to borrow English or Mandarin corpora; a dedicated Spanish resource now exists for training and benchmarking naturalness predictors.
  • Fine-tuning an English-trained quality model on Spanish data improves its accuracy, reinforcing the idea that MOS prediction benefits from in-language adaptation.
  • Comparable errors from a compact network on frozen self-supervised features suggest that naturalness prediction can be done with modest computational budgets.
  • The system-grouped rating analysis provides a reusable quality map of current Spanish TTS systems, useful for practitioners choosing synthesis backends.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because most audio clips received a single rating, the reported MAE likely measures agreement with imputed labels rather than with true averaged human opinion; collecting multiple ratings per clip on a subset would quantify the ceiling of the model.
  • The underrepresentation of scores near 4 identified in the paper suggests the dataset is least reliable near the boundary between high and acceptable quality; oversampling that range in a future release could improve calibration.
  • The same DenseMOS recipe could be applied to other languages or to speech-quality dimensions beyond naturalness, such as intelligibility or speaker similarity, to test whether the preference for local CNN features is a general property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a new Spanish-language corpus for evaluating TTS naturalness: 4,326 audio samples from a range of synthetic and human voices, rated through a listening test inspired by ITU-T Rec. P.807, with 92 participants. The authors fine-tune NISQA v1.0 on this corpus and also train two small DenseMOS predictors on frozen wav2vec 2.0 embeddings. On a held-out split with no overlapping speakers or systems, the fine-tuned NISQA reaches a mean absolute error of 0.81 and the DenseMOS variants reach 0.80–0.81, which the paper argues is comparable to English-language work. The paper claims this is the first dataset of its kind for Spanish TTS quality assessment.

Significance. If the per-sample labels are reliable, this is a useful and timely contribution: it provides a Spanish-language MOS-style corpus, follows a documented subjective-testing protocol, enforces speaker/system-disjoint splits, and makes code and data publicly available. The use of an external English-trained baseline and two independent modeling routes strengthens the validation design. The main caution is that the dataset is largely single-rated per audio clip and the participant pool is almost entirely Argentine, so the strength of the central per-audio prediction claim is not yet established. These issues are fixable but must be addressed before the reported MAE values can be taken as evidence of per-sample naturalness prediction quality.

major comments (3)
  1. [§2.2, §4.1] The dataset's use as training and test ground truth depends on per-audio MOS labels that, according to §2.2, are mostly single ratings. The agreement metrics in §4.1 are computed after grouping by TTS system or speaker and after mean-imputing missing ICC values; this measures group-level consistency, not per-clip label reliability, and can remain high even if individual clip labels are very noisy. Because the reported MAE of 0.80–0.81 is computed against those labels, the main validation claim is not currently supported. Please report the exact distribution of ratings per audio clip, provide per-clip agreement measures on the multi-rated subset, and either collect additional ratings for at least the test set or reframe the evaluation as prediction of system-level or speaker-level means with appropriate uncertainty.
  2. [§2.2, §3.1, §5] The reported counts are internally inconsistent: the text states 4,326 ratings, the split in §3.1 sums to 3,924, the introduction states 4,236 speech samples, and the abstract says 52 different TTS systems and human voices, while Table 1 lists 12 TTS systems plus human speech and §5 says 52 different speakers. Clarify whether these numbers refer to unique audio files, ratings, systems, or speakers, explain the 402-item gap between the total and the train/validation/test sum, and correct the inconsistencies.
  3. [Table 2, §4.2] The table caption promises 95% confidence intervals, but none are reported. The headline comparisons are small (PCC 0.71 vs 0.73, MAE 0.81 vs 0.80) and may well be within noise. Add bootstrap or permutation confidence intervals for PCC, MAE, and RMSE, and state the number of test items per system; without this, the claim that the models are comparable to English-language benchmarks cannot be evaluated.
minor comments (4)
  1. [Table 2] The table uses decimal commas (0,71, 0,99); the text should use decimal points for consistency with standard English-language scientific notation.
  2. [§2.2] The response-validity criterion that discards a participant who rates a human voice equal to a data-augmented sample should be described in more detail, including whether this was applied per pair, per participant, or per rating, and how many participants were removed entirely versus how many individual ratings were excluded.
  3. [§1, §5, Abstract] The paper should explicitly acknowledge that the human reference voices and most participants are from Argentina, so the corpus is primarily Rioplatense Spanish rather than a fully pan-Spanish resource; this should be stated in the abstract and conclusions to avoid overgeneralization.
  4. [§4.1] The description of the agreement computation should specify the exact unit of analysis (ratings per system, per speaker, or per audio) and the number of groups used in the ICC and Krippendorff's Alpha calculations, since the current presentation makes the group-level nature of these metrics easy to overlook.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset, model training, and held-out evaluation are self-contained and empirically grounded.

full rationale

The paper's central claims are the construction of a Spanish TTS naturalness dataset and the validation of automatic MOS predictors trained on it. The derivation chain is ordinary supervised machine learning: subjective MOS labels were collected via a listening test, models (NISQA fine-tuned and DenseMOS variants) were trained on a training split, and performance was measured on a held-out test split with no speaker or TTS-system overlap. The baseline NISQA v1.0 is an external English-trained model, and the reported improvement after fine-tuning is an empirical result, not a consequence of construction. The DenseMOS architecture is inspired by prior work, including a self-citation to Pepino et al. [25], but that citation only motivates the use of wav2vec 2.0 embeddings and a weighted averaging layer; it does not supply the dataset labels, the trained weights, or the test-set numbers. The paper's own limitation statements about single-rating labels, label imbalance around MOS 4, and mean-biased predictions are data-quality observations, not circular reasoning. No equation in the paper defines a prediction in terms of the fitted quantity, and no parameter is fitted to the test set and then reported as a prediction. Accordingly, no specific circular step can be quoted, and the correct finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is a dataset, not a physical theory, so no new particles, forces, or mediators are introduced. The free parameters listed are model hyperparameters and augmentation choices that affect the empirical results.

free parameters (3)
  • DenseMOS hyperparameters = hidden size 128, dropout 0.6, learning rates 0.001 and 0.0001, early stopping patience 40
    Chosen by hand based on validation performance (§3.2); Table 2 results depend on these choices.
  • NISQA fine-tuning settings = learning rate 0.001, ADAM, MSE loss
    Taken from prior literature and the model repository (§3.1); not derived within the paper.
  • VTLP deformation factor range = random between 0.9 and 1.1
    Selected following Jaitly et al. (§2.1); this choice shapes the augmented stimuli in the dataset.
assumptions (4)
  • domain assumption ITU-T Rec. P.807 subjective test methodology yields valid MOS ratings for TTS naturalness.
    The entire labeling procedure follows this standard (§2.2). If the self-selected number of stimuli violates the protocol, labels may be biased.
  • domain assumption Audio from the Argentinian Spanish dataset SLR61 is representative for generating diverse TTS stimuli.
    All synthetic sentences and human reference samples come from this single corpus (§2.1), limiting coverage of Spanish varieties.
  • ad hoc to paper A single rating per audio clip is adequate ground truth for training and evaluating naturalness predictors.
    The paper states 'Most of the audios received a single rating' (§2.2) and uses these labels directly for model training; this is a strong assumption given MOS variability.
  • domain assumption Frozen wav2vec 2.0 representations contain enough information for naturalness prediction.
    DenseMOS relies entirely on wav2vec 2.0 embeddings (§3.2); if these features lack quality-related information, the downstream network cannot recover it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Dataset for Automatic Assessment of TTS Quality in Spanish." pith.science (2026). https://pith.science/paper/YN5DWTKE

@misc{pith2026250701805,
  author       = {Pith},
  title        = {Pith review of: A Dataset for Automatic Assessment of TTS Quality in Spanish},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YN5DWTKE}},
  note         = {Machine review of arXiv:2507.01805}
}
read the original abstract

This work addresses the development of a database for the automatic assessment of text-to-speech (TTS) systems in Spanish, aiming to improve the accuracy of naturalness prediction models. The dataset consists of 4,326 audio samples from 52 different TTS systems and human voices and is, up to our knowledge, the first of its kind in Spanish. To label the audios, a subjective test was designed based on the ITU-T Rec. P.807 standard and completed by 92 participants. Furthermore, the utility of the collected dataset was validated by training automatic naturalness prediction systems. We explored two approaches: fine-tuning an existing model originally trained for English, and training small downstream networks on top of frozen self-supervised speech models. Our models achieve a mean absolute error of 0.8 on a five-point MOS scale. Further analysis demonstrates the quality and diversity of the developed dataset, and its potential to advance TTS research in Spanish.

Figures

Figures reproduced from arXiv: 2507.01805 by the authors.

Figure 1
Figure 1. MOS received for each evaluated system in the subjective test. The model names correspond to the ones presented in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Weights of the trainable weighted average layer of DenseMOS and DenseMOS-960h models blocks, the learned representations become increasingly aligned with the original training and fine-tuning tasks of wav2vec 2.0, which were primarily designed for ASR. Despite their architec￾tural differences, NISQA and DenseMOS achieved compara￾ble performance, suggesting that CNN-based architectures are well-suited for capturing t… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 32 canonical work pages

  1. [1]

    They are widely used in virtual voice assistants such as Siri, Gemini and Alexa, as well as in accessibility applications designed to as- sist visually impaired individuals

    Introduction In recent years, voice synthesis technologies have become in- creasingly integrated into everyday human activities. They are widely used in virtual voice assistants such as Siri, Gemini and Alexa, as well as in accessibility applications designed to as- sist visually impaired individuals. Additionally, they play a sig- nificant role in entert...

    work page

  2. [2]

    A Dataset for Automatic Assessment of TTS Quality in Spanish

    Data Collection and Evaluation 2.1. Data Collection For data collection, we considered TTS systems spanning di- verse synthesis techniques and speech quality levels (Table 1). Some systems allowed selecting different speakers or control- ling their characteristics, like gender, accent and regional di- alect. Using these controls, we generated 40 unique sp...

    work page Pith review arXiv 2025

  3. [3]

    Fine-tuning NISQA Based on the subjective test responses, we fine-tuned NISQA v1.02 for Spanish TTS naturalness assessment

    Model design 3.1. Fine-tuning NISQA Based on the subjective test responses, we fine-tuned NISQA v1.02 for Spanish TTS naturalness assessment. The model pro- 2https://github.com/gabrielmittag/NISQA cesses mel-spectrograms, segmented into 150 ms windows with a 10 ms hop length. These are passed through a CNN and then fed into a LSTM network to capture tempo...

    work page

  4. [4]

    Dataset meta-analysis A total of 4,326 audio samples were rated in the subjective sur- vey, amounting to 226 minutes of speech data

    Results and discussion 4.1. Dataset meta-analysis A total of 4,326 audio samples were rated in the subjective sur- vey, amounting to 226 minutes of speech data. Each speaker is associated with a gender (male or female) and a dialect. The most represented dialect is Rioplatense Spanish with 23 voices, followed by Castilian Spanish (9 voices) and Central Am...

    work page

  5. [5]

    Gender and voice number are indicated, and VTLP and GL suffixes specify if the samples were generated through data augmentation

    If multiple tiers of quality were available for a given model, the letter P indicates the version with the advertised higher quality. Gender and voice number are indicated, and VTLP and GL suffixes specify if the samples were generated through data augmentation. Figure 2: Weights of the trainable weighted average layer of DenseMOS and DenseMOS-960h models...

    work page

  6. [6]

    The collected dataset consists of 4,326 audio samples from 52 different speakers (12 different TTS systems and 6 human voices)

    Conclusions This study focused on the development and validation of a dataset for the automatic evaluation of TTS systems in Span- ish. The collected dataset consists of 4,326 audio samples from 52 different speakers (12 different TTS systems and 6 human voices). The samples span a wide range of speech quality, di- alects, and speaker genders. A subjectiv...

    work page

  7. [7]

    P.800.1, 2016

    Mean Opinion Score (MOS) Terminology, International Telecom- munication Union Std. P.800.1, 2016

    work page 2016

  8. [8]

    Analysis of the survey of voice synthesis technology,

    Z. Zhou, “Analysis of the survey of voice synthesis technology,” Applied and Computational Engineering, 2023

    work page 2023

Show all 34 references
  1. [9]

    Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in ICASSP, 2001

    2001

  2. [10]

    Perceptual objective listening quality assessment (polqa), the third generation itu-t standard for end-to- end speech quality measurement part i-temporal alignment,

    J. Beerends, C. Schmidmer, J. Berger, M. Obermann, R. Ullmann, J. Pomy, and M. Keyhl, “Perceptual objective listening quality assessment (polqa), the third generation itu-t standard for end-to- end speech quality measurement part i-temporal alignment,”AES, 2013

    2013

  3. [11]

    Deep learning based assessment of syn- thetic speech naturalness,

    G. Mittag and S. M ¨oller, “Deep learning based assessment of syn- thetic speech naturalness,” in Interspeech, 2020

    2020

  4. [12]

    Noresqa: A framework for speech quality assessment using non-matching references,

    P. Manocha, B. Xu, and A. Kumar, “Noresqa: A framework for speech quality assessment using non-matching references,” arXiv:2109.08125, 2021

    2021 arXiv

  5. [13]

    Amazon Polly – Text to Speech in 47 V oices and 24 Languages,

    Amazon, “Amazon Polly – Text to Speech in 47 V oices and 24 Languages,” accessed: 2025-02-

    2025

  6. [14]

    Dctts: Discrete diffusion model with contrastive learning for text-to-speech generation,

    Z. Wu, Q. Li, S. Liu, and Q. Yang, “Dctts: Discrete diffusion model with contrastive learning for text-to-speech generation,” in ICASSP, 2024

    2024

  7. [15]

    Microsoft Azure Speech Services,

    Microsoft, “Microsoft Azure Speech Services,” accessed: 2025- 02-17. [Online]. Available: https://azure.microsoft.com

    2025

  8. [16]

    [Online]

    Speechelo, accessed: 2025-02-17. [Online]. Available: https: //speechelo.com

    2025

  9. [17]

    Available: https://aws.amazon.com/blogs/aws/ polly-text-to-speech-in-47-voices-and-24-languages/

    [Online]. Available: https://aws.amazon.com/blogs/aws/ polly-text-to-speech-in-47-voices-and-24-languages/

  10. [18]

    Servicio de Conversi ´on de Texto a Habla Hispanoamericano,

    Neurasound, “Servicio de Conversi ´on de Texto a Habla Hispanoamericano,” accessed: 2025-02-17. [Online]. Available: https://www.neurasound.com

    2025

  11. [19]

    Texto a V oz de Loquendo Online,

    Loquendo, “Texto a V oz de Loquendo Online,” ac- cessed: 2025-02-17. [Online]. Available: https://loquendo. online/texto-a-voz-de-loquendo-online

    2025

  12. [20]

    text-to-speech 1.6.1,

    Thomas Dewitte, “text-to-speech 1.6.1,” accessed: 2025-02-17. [Online]. Available: https://pypi.org/project/text-to-speech

    2025

  13. [21]

    Fastpitch: Parallel text-to-speech with pitch predic- tion,

    A. Ła ´ncucki, “Fastpitch: Parallel text-to-speech with pitch predic- tion,” in ICASSP, 2021

    2021

  14. [22]

    Speech synthesis method based on tacotron2,

    Y . Li, D. Qin, and J. Zhang, “Speech synthesis method based on tacotron2,” in ICACI, 2021

    2021

  15. [23]

    Crowdsourcing Latin American Spanish for Low-Resource Text- to-Speech,

    A. Guevara-Rukoz, I. Demirsahin, F. He, S.-H. C. Chu, S. Sarin, K. Pipatsrisawat, A. Gutkin, A. Butryna, and O. Kjartansson, “Crowdsourcing Latin American Spanish for Low-Resource Text- to-Speech,” in LREC, 2020

    2020

  16. [24]

    Single-channel waveforms augmentations for speech recognition models

    Darya V ozhdaeva, “Single-channel waveforms augmentations for speech recognition models.” accessed: 2025-02-17. [Online]. Available: https://github.com/waveletdeboshir/speechaugs

    2025

  17. [25]

    V ocal tract length perturbation (vtlp) improves speech recognition,

    N. Jaitly and G. E. Hinton, “V ocal tract length perturbation (vtlp) improves speech recognition,” inICML Workshop on Deep Learn- ing for Audio, Speech and Language, 2013

    2013

  18. [26]

    P.807, 2016

    Subjective test methodology for assessing speech intelligibility , International Telecommunication Union Std. P.807, 2016

    2016

  19. [27]

    Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,

    G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,” in Interspeech, 2021

    2021

  20. [28]

    Bias-aware loss for training image and speech quality prediction models from multiple datasets,

    G. Mittag, S. Zadtootaghaj, T. Michael, B. Naderi, and S. Moller, “Bias-aware loss for training image and speech quality prediction models from multiple datasets,” in QoMEX, 2021

    2021

  21. [29]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv:1904.05862, 2019

    1904 arXiv

  22. [30]

    A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recogni- tion, speaker verification and spoken language understanding,

    Y . Wang, A. Boumadane, and A. Heba, “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recogni- tion, speaker verification and spoken language understanding,” arXiv:2111.02735, 2022

    2022 arXiv

  23. [31]

    On the use of self-supervised pre-trained acoustic and linguistic features for continuous speech emotion recognition,

    M. Macary, M. Tahon, Y . Est `eve, and A. Rousseau, “On the use of self-supervised pre-trained acoustic and linguistic features for continuous speech emotion recognition,” arXiv:2011.09212, 2020

    2011 arXiv

  24. [32]

    Emotion recognition from speech using wav2vec 2.0 embeddings,

    L. Pepino, P. Riera, and L. Ferrer, “Emotion recognition from speech using wav2vec 2.0 embeddings,” in Interspeech, 2021

    2021

  25. [33]

    Recognizing more emotions with less data using self-supervised transfer learning,

    J. Boigne, B. Liyanage, and T. ¨Ostrem, “Recognizing more emotions with less data using self-supervised transfer learning,” arXiv:2011.05585, 2020

    2011 arXiv

  26. [34]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville,Deep Learning. MIT Press, 2016, chapter 8

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.