REVIEW 4 major objections 5 minor 24 references
Learning to assess subjective impressions from speech
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that pairwise comparison labels train a voice-assessing model better than absolute ratings, even with very little data.
desk verdict New task and dataset for subjective voice descriptor scoring, but the headline CCR-beats-ACR result rests on test-set-guided epoch selection and needs a validation-based check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is RankNet applied to CCR labels: a network takes two speech representations, produces scores for each, and is trained with binary cross-entropy on the sigmoid of their score difference, with the four CCR response options mapped to target probabilities 0, 0.25, 0.75, and 1. The evaluation metric ppref (precision of preferences) is the accuracy of the predicted score ordering on held-out CCR pairs, with separate strong and weak variants. This machinery carries the argument because it lets a model learn a scalar score from pairwise comparisons alone, and the metric directly measures whether the learned score reproduces human preference orderings.
What would settle it
Run the same experiment with early stopping on a held-out validation set rather than the test set; if ACR training then matches or exceeds CCR training in ordering accuracy, the paper's main claim is falsified. Alternatively, on a subjective attribute where absolute ratings are known to be highly reliable and pairwise comparisons are noisy, ACR training would be expected to win, which would refute the claimed universal superiority of CCR training.
Extended reading notes
Core claim
The central discovery is that comparison category ratings, though collected on a small random subset of all possible pairs, teach a score-predicting network to order voices by a subjective descriptor more accurately than an equal-sized set of absolute ratings. This holds for three descriptors (youthful female voice, youthful male voice, resonant male voice) and two architectures: a MOSNet-style CNN-BLSTM-FC and a fixed wav2vec2.0 feature extractor with a small trainable head. The authors attribute the advantage to the higher reliability of pairwise comparison labels, since listeners agree more readily on which of two voices is 'more' something than on assigning an absolute number. They also find that ppref-strong, computed on confident preference labels, improves with more training data and exceeds ppref-weak, computed on 'a little more' labels, suggesting that low-confidence comparisons are noisy and should be filtered out when evaluating performance.
Load-bearing premise
The paper's headline comparison comes from selecting, for each random initialization, the training epoch with the best test-set ordering accuracy, so the reported advantage of CCR training over ACR training could depend on test-set information leaking into model selection.
Editorial extensions
If this is right
- A single user's pairwise 'which is more X' judgments, even numbering in the hundreds, can ground a usable voice-impression scorer for that user.
- For subjective voice descriptor assessment, CCR-based training should be preferred over ACR-based regression, especially in the low-data regime where personalized models operate.
- The ppref-strong metric is a suitable primary evaluation for such models, and weak 'a little more' comparisons should be discarded or down-weighted in evaluation.
- The data-collection protocol and training method extend beyond fixed descriptors to personalized descriptors like 'my favorite voice' because they tolerate very small label budgets.
Reading between the lines
- The paper selects, for each random initialization, the training epoch with the highest test-set ordering accuracy; an independent replication with validation-based early stopping would reveal how much of the CCR-over-ACR advantage survives without test-set peeking.
- The same CCR-versus-ACR comparison could be tested in other subjective domains, such as image aesthetics or product review helpfulness, where pairwise preferences are also easier to collect than absolute scores.
- Because CCR labels appear more reliable, a multi-task loss combining CCR and ACR labels might push ordering accuracy beyond CCR alone, a combination the paper does not test.
- The weak 'a little more' labels contributed little signal in this study, suggesting that a forced two-option choice without intensity distinctions could be a more label-efficient collection strategy for personalized models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new task of training neural network models to assess subjective impressions of speech (referred to as subjective voice descriptors, SVDs), with a focus on personalized descriptors such as 'my favorite voice.' The authors construct a dataset containing both absolute category ratings (ACR) and comparison category ratings (CCR) for three SVDs (youthful female, youthful male, resonant male). They propose a CCR-based evaluation metric called ppref (precision of preferences), and compare two training methods: ACR-based MSE training and CCR-based RankNet training. The empirical claim is that CCR training is superior to ACR training, and that moderate ppref can be achieved even with very limited training data, supporting the feasibility of personalized SVD assessment models.
Significance. If the central empirical claim is supported by a sound evaluation, the paper makes a useful contribution by proposing a new task, a new dataset, and a practical training strategy based on pairwise comparisons. The finding that CCR labels can be learned from small amounts of data would be valuable for personalized speech assessment. However, the current evaluation protocol has a load-bearing flaw: the reported ppref values are the maximum over training epochs selected directly on the test set, which inflates the measured performance and can bias the comparison between methods with different training dynamics. The paper also evaluates only on a metric that aligns with the CCR training objective. These issues must be addressed before the quantitative conclusions can be considered reliable.
major comments (4)
- [VI-A, VI-B, Figure 4] The reported ppref values are computed as the maximum over all 30 training epochs, evaluated on the same test set used for the final assessment, for each of the five random initializations (Section VI-A: 'The training was run for 30 epochs'; Section VI-B: 'maximum pprefs over all epochs'). No validation split or early-stopping rule is described. This test-set-guided epoch selection biases the reported numbers upward, and because RankNet (CCR) and MSE (ACR) training converge at different rates and have different epochwise variance, the max-over-epoch comparison can systematically favor the method whose epochwise curve happens to peak higher even if the methods perform equally at a principled stopping point. The claim that 'CCR training is superior to ACR training' is therefore not supported by the current protocol. Please repeat the comparison using model selection on a validation split (e.g., select the epoch with best validation ppref) or report per-epoch curves and a comparison at a fixed training duration.
- [IV-A, V-B] The only evaluation metric, ppref, is computed from CCR labels (Section IV-A), which are exactly the labels used in the CCR training objective (Equation 1). ACR training, in contrast, minimizes MSE on five-point ACR scores. Evaluating ACR-trained models on ppref conflates 'better at predicting CCR preferences' with 'better assessment model,' giving CCR training a built-in advantage. The paper should include an ACR-based evaluation metric (e.g., MSE or rank correlation on held-out ACR labels) or otherwise justify that ppref is the appropriate single metric for comparing the two training approaches.
- [III] The SVDs used in the experiments were deliberately pre-selected for low inter-rater variance and relatively large pseudo F values (Section III: 'We deliberately selected SVDs that showed relatively little variation in opinion among participants'). The upper-bound estimates in Table 1 are derived from the same annotator population. This selection biases the task toward easier SVDs, so the claim that 'moderate ppref is achievable with very limited data' may not generalize to SVDs with higher inter-rater disagreement. Please discuss this selection bias and, if possible, report results on at least one SVD with higher variance to assess the robustness of the findings.
- [VI-B] The claim that 'CCR training is superior to ACR training' is based on only five random initializations, with no significance testing reported. In Figure 4, the standard deviations appear to overlap for several training-data conditions (e.g., fixedSSL-FC at 5000 samples), so the observed differences may not be statistically reliable. Please report paired significance tests (e.g., Wilcoxon signed-rank test across the five trials) or confidence intervals for the difference in ppref between the two training methods at each training-data size.
minor comments (5)
- [Abstract] There is a typo: 'abosolute' should be 'absolute.'
- [V-B] The BCE loss equation is missing a negative sign and a sum over pairs: it should be L = -Σ[ P_ij log(P̂_ij) + (1-P_ij) log(1-P̂_ij) ]. Please correct the formula.
- [IV-A] The capitalization of 'ppref' is inconsistent (e.g., 'Ppref' in Table 1 header and 'ppref' in the text). Please standardize.
- [III] The phrase 'preliminary casual experiment' is vague; please provide a brief description of how the pseudo F values were computed and how the selection threshold was determined.
- [VI-A] The description of the test-set construction is terse: 'the test data are a portion of CCRs assigned to the speaker pair including at least one speaker not included in Strain.' Please clarify how the test pairs were sampled and whether any speakers overlap between training and test pairs.
Circularity Check
No circularity: the CCR-vs-ACR comparison is an empirical held-out evaluation, not a derivation from the metric or from self-citations.
full rationale
The paper's central claims are that ppref is moderate with limited training data and that CCR training outperforms ACR training. These are empirical findings, not consequences of the definitions. ppref is computed on held-out CCR test pairs that include at least one speaker absent from the training set, and no model parameter is fitted to ppref. The CCR training objective (RankNet binary cross-entropy on CCR labels) and the ppref metric both involve pairwise score orderings, so the metric is aligned with the CCR training approach, but this alignment does not make the outcome true by construction: the model must still generalize to unseen speakers and unseen CCR pairs, and ACR training could in principle achieve high ppref if ACR scores track the underlying ordering. The upper-bound estimates use a separate set of 50 common CCR questions from the same participants but not the speech samples used in the main experiments, and they are not used to train or select models. There are no load-bearing self-citations and no imported uniqueness theorems; the use of existing architectures (MOSNet, wav2vec2.0) and the ppref metric from Carterette et al. are appropriately attributed. The reported selection of the maximum ppref over all 30 epochs on the test set is a methodological limitation that could inflate the absolute ppref values and potentially bias the CCR-vs-ACR comparison, but test-set-guided epoch selection is an evaluation flaw rather than a circular reduction of the result to its inputs. No equation in the paper is equivalent by construction to another, and no fitted parameter is renamed as a prediction. Thus no circularity is present; the main concerns belong to experimental validity rather than circular reasoning.
Assumptions & free parameters
free parameters (4)
- learning rate =
0.0001
- dropout rate =
0.3
- batch size =
6
- CCR label mapping =
0, 0.25, 0.75, 1
assumptions (4)
- domain assumption ACR and CCR labels aggregated from 50 native Japanese annotators reflect the subjective impression of the target population.
- domain assumption Pretrained wav2vec2.0 features (LibriSpeech) and magnitude spectrograms carry sufficient information to predict subjective voice descriptors.
- domain assumption The estimated upper bound for ppref, based on average participant agreement on 50 common questions, is a valid proxy for the achievable ceiling.
- domain assumption CCR labels are more reliable than ACR labels (as asserted in [7], [8]), and this reliability advantage carries over to SVD assessment.
Cite this review
Pith. "Pith review of Learning to assess subjective impressions from speech." pith.science (2026). https://pith.science/paper/RBFWVJWA
@misc{pith2026250619335,
author = {Pith},
title = {Pith review of: Learning to assess subjective impressions from speech},
year = {2026},
howpublished = {\url{https://pith.science/paper/RBFWVJWA}},
note = {Machine review of arXiv:2506.19335}
}
read the original abstract
We tackle a new task of training neural network models that can assess subjective impressions conveyed through speech and assign scores accordingly, inspired by the work on automatic speech quality assessment (SQA). Speech impressions are often described using phrases like `cute voice.' We define such phrases as subjective voice descriptors (SVDs). Focusing on the difference in usage scenarios between the proposed task and automatic SQA, we design a framework capable of accommodating SVDs personalized to each individual, such as `my favorite voice.' In this work, we compiled a dataset containing speech labels derived from both abosolute category ratings (ACR) and comparison category ratings (CCR). As an evaluation metric for assessment performance, we introduce ppref, the accuracy of the predicted score ordering of two samples on CCR test samples. Alongside the conventional model and learning methods based on ACR data, we also investigated RankNet learning using CCR data. We experimentally find that the ppref is moderate even with very limited training data. We also discover the CCR training is superior to the ACR training. These results support the idea that assessment models based on personalized SVDs, which typically must be trained on limited data, can be effectively learned from CCR data.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A. Rix, J. G Beerends, M. P Hollier, and A. P Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in Proc. IEEE ICASSP , 2001
work page 2001
-
[2]
MOSNet: Deep learning based objective assessment for voice conversion,
C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y . Tsao, and H. M. Wang, “MOSNet: Deep learning based objective assessment for voice conversion,” in Proc. Interspeech, 2019
work page 2019
-
[3]
Speech quality assessment through MOS using non-matching references,
P. Manocha and A. Kumar, “Speech quality assessment through MOS using non-matching references,” in Proc. Interspeech, 2022
work page 2022
-
[4]
Personalized Audio Quality Preference Prediction
C.-C. Wang, Y .-C. Lin, Y .-T. Hsu, and J.-S. R. Jang, “Personalized Audio Quality Preference Prediction,” arXiv preprint arXiv:2302.08130 , 2023
work page Pith review arXiv 2023
-
[5]
Generalization ability of MOS prediction networks,
E. Cooper, W.-C. Huang, T. Toda, and J. Yamagishi, “Generalization ability of MOS prediction networks,” in Proc. IEEE ICASSP . IEEE, 2022, pp. 8442–8446
work page 2022
-
[6]
Methods for subjective determination of transmis- sion quality,
ITU-T Rec. P. 800, “Methods for subjective determination of transmis- sion quality,” 1996
work page 1996
-
[7]
Here or There: Preference Judgments for Relevance,
B. Carterette, P. N Bennett, D. M. Chickering, and S. T Dumais, “Here or There: Preference Judgments for Relevance,” in Proc. ECIR. Springer, 2008, pp. 16–27
work page 2008
-
[8]
Comparison of four subjective methods for image quality assessment,
R. K Mantiuk, A. Tomaszewska, and R. Mantiuk, “Comparison of four subjective methods for image quality assessment,” in Computer graphics forum. Wiley Online Library, 2012, vol. 31, pp. 2478–2491
work page 2012
Show all 24 references
-
[9]
AutoMOS: Learning a non-intrusive assessor of naturalness- of-speech,
B. Patton, Y . Agiomyrgiannakis, M. Terry, K. Wilson, R. A. Saurous, and D. Sculley, “AutoMOS: Learning a non-intrusive assessor of naturalness- of-speech,” in Proc. NIPS End-to-end Learning for Speech and Audio Processing Workshop, 2016
2016
-
[10]
MBNet: MOS prediction for synthesized speech with mean-bias network,
Y . Leng, X. Tan, S. Zhao, F. Soong, X.-Y . Li, and T. Qin, “MBNet: MOS prediction for synthesized speech with mean-bias network,” in Proc. IEEE ICASSP . IEEE, 2021, pp. 391–395
2021
-
[11]
LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech,
W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech,” in Proc. IEEE ICASSP . IEEE, 2022, pp. 896–900
2022
-
[12]
UTMOS: Utokyo-Sarulab System for V oiceMOS Challenge 2022,
T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: Utokyo-Sarulab System for V oiceMOS Challenge 2022,” in Proc. Interspeech, 2022, pp. 4521–4525
2022
-
[13]
MOSPC: MOS Prediction Based on Pairwise Comparison,
K. Wang, Y . Zhao, Q. Dong, T. Ko, and M. Wang, “MOSPC: MOS Prediction Based on Pairwise Comparison,” arXiv preprint arXiv:2306.10493, 2023
2023 arXiv
-
[14]
Preference-based training framework for automatic speech quality assessment using deep neural network,
C.-H. Hu, Y . Yasuda, and T. Toda, “Preference-based training framework for automatic speech quality assessment using deep neural network,” arXiv preprint arXiv:2308.15203 , 2023
2023 arXiv
-
[15]
PromptTTS: Controllable text-to-speech with text descriptions,
Z. Guo, Y . Leng, Y . Wu, S. Zhao, and X. Tan, “PromptTTS: Controllable text-to-speech with text descriptions,” in Proc. IEEE ICASSP . IEEE, 2023, pp. 1–5
2023
-
[16]
Coco-Nut: Corpus of Japanese Utterance and V oice Characteristics Description for Prompt-Based Control,
A. Watanabe, S. Takamichi, Y . Saito, W. Nakata, D. Xin, and H. Saruwatari, “Coco-Nut: Corpus of Japanese Utterance and V oice Characteristics Description for Prompt-Based Control,” in Proc. IEEE ASRU. IEEE, 2023, pp. 1–8
2023
-
[17]
StyleCap: Automatic Speaking- Style Captioning from Speech Based on Speech and Language Self- supervised Learning Models,
K. Yamauchi, Y . Ijima, and Y . Saito, “StyleCap: Automatic Speaking- Style Captioning from Speech Based on Speech and Language Self- supervised Learning Models,” arXiv preprint arXiv:2311.16509 , 2023
2023 arXiv
-
[18]
Extraction of everyday expression associated with voice quality of normal utterance,
H. Kido and H. Kasuya, “Extraction of everyday expression associated with voice quality of normal utterance,” J. ASJ, 1999, in Japanese
1999
-
[19]
A dendrite method for cluster analysis,
T. Cali ´nski and J. Harabasz, “A dendrite method for cluster analysis,” Commun. Stat.-theory Methods , 1974
1974
-
[20]
The proof and measurement of association between two things.,
C. Spearman, “The proof and measurement of association between two things.,” 1961
1961
-
[21]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12449– 12460, 2020
2020
-
[22]
Librispeech: An ASR corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. IEEE ICASSP. IEEE, 2015, pp. 5206–5210
2015
-
[23]
fairseq: A fast, extensible toolkit for sequence modeling,
M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in Proc. NAACL-HLT: Demonstrations, 2019
2019
-
[24]
Learning to Rank Using Gradient Descent,
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to Rank Using Gradient Descent,” in Proc. ICML, 2005, pp. 89–96
2005
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.