{"id":"f52876bd-8e02-4a55-97b7-d9c0015313e8","arxiv_id":"2506.19335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Models trained on comparison category ratings (CCR) achieve better preference-ranking accuracy than models trained on absolute category ratings (ACR) for subjective voice descriptor scoring.","lead":"This paper introduces a new task: training neural networks to score speech based on subjective voice descriptions like 'cute voice', using both absolute ratings and pairwise comparisons. It finds that models trained on pairwise comparison labels outperform those trained on absolute scores, even with small amounts of data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set-guided epoch selection likely inflates reported ppref; the CCR-over-ACR gap needs a validation-based check.","rationale":"I read the paper as making two connected empirical claims: (1) moderate ppref is achievable with limited CCR/ACR data, and (2) CCR training is superior to ACR training, supporting CCR as the preferred data format for personalized SVD assessment. The design of the experiment is largely sound in other respects: speaker-disjoint test splits, multiple initializations, and separate training/test data are all present. The weakest link is the evaluation protocol itself. Section VI-A says the model is trained for 30 epochs, and Figure 4 plots the averaged value of maximum pprefs over all epochs. Selecting the best epoch per run on the test set is a form of test-set model selection. The paper never states that the selected epochs are consistent across runs or methods, nor that the max is stable. This matters specifically for the comparative claim: the gap between CCR and ACR training could be an artifact of different epochwise dynamics. ACR (MSE on 5-point ratings) is a smoother, lower-variance objective; RankNet on pairwise labels may keep improving ppref later in training or fluctuate more across epochs, making a max-over-epoch protocol differentially favorable to CCR. The metric-alignment concern reinforces this: ppref is computed from CCR preferences, and CCR training directly optimizes a smooth surrogate of that quantity (Equation 1 with Pi,j in {0, 0.25, 0.75, 1}), while ACR training only optimizes it indirectly through the latent score. So the comparison is partly circular. I do not think this requires rejection: the task and dataset are novel, the upper-bound estimates in Table I are a useful sanity check, and the CCR advantage is reported consistently across all SVDs and both architectures. But the magnitude and even the sign of the advantage are not established until the epoch-selection bias is removed. The reader's CONDITIONAL verdict is appropriate, and the concrete validation-based re-analysis I propose would settle whether the stated claims hold. I agree with the reader's identification of test-set-guided epoch selection as the load-bearing weakness, and I would add that the metric-alignment issue is a secondary amplifier rather than an independent fatal flaw.","tokens_in":7926,"tokens_out":1888,"duration_ms":20680,"concrete_test":"Re-run the three SVDs with both architectures (or, minimally, youthfulF with CNN-BLSTM-FC and fixedSSL-FC) using a held-out validation split to select the epoch (e.g., highest ppref-strong on validation CCRs, or a fixed 10-epoch rule), then report test ppref-strong and ppref-weak for that single epoch per run. If the CCR-minus-ACR gap shrinks by more than a few points or changes sign under this protocol, the headline superiority claim would not survive in its current form.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim (CCR training outperforms ACR training, and moderate ppref is attainable with very limited data) rests on Figure 4, which reports the maximum ppref over all 30 training epochs, selected directly on the CCR test set, for each of five initializations (Section VI-A, VI-B). No validation split is used, and no early-stopping or model-selection rule is specified. Because ppref on a fixed test set fluctuates across epochs, taking the per-run maximum over epochs is a selection procedure that biases the reported number upward, and the bias can differ between training methods. RankNet (CCR) and MSE (ACR) converge at different rates and have different epochwise variance profiles; if ACR training peaks earlier or fluctuates more, the max-over-epoch comparison systematically favors CCR even when the methods are equal or ACR is better at a principled stopping point. A second, compounding issue is that the single evaluation metric ppref is computed from CCR labels (Section IV-A), which is exactly the objective CCR training optimizes (Equation 1), so the comparison conflates 'better at the training objective' with 'better assessment model.' The paper explicitly frames personalized SVD use cases (Section I criterion (1)), where a single user's limited labels must be used without test-set peeking; the reported protocol does not match that deployment scenario.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task of training neural network models to assess subjective impressions of speech (referred to as subjective voice descriptors, SVDs), with a focus on personalized descriptors such as 'my favorite voice.' The authors construct a dataset containing both absolute category ratings (ACR) and comparison category ratings (CCR) for three SVDs (youthful female, youthful male, resonant male). They propose a CCR-based evaluation metric called ppref (precision of preferences), and compare two training methods: ACR-based MSE training and CCR-based RankNet training. The empirical claim is that CCR training is superior to ACR training, and that moderate ppref can be achieved even with very limited training data, supporting the feasibility of personalized SVD assessment models.","tokens_in":8248,"tokens_out":4187,"duration_ms":43439,"significance":"If the central empirical claim is supported by a sound evaluation, the paper makes a useful contribution by proposing a new task, a new dataset, and a practical training strategy based on pairwise comparisons. The finding that CCR labels can be learned from small amounts of data would be valuable for personalized speech assessment. However, the current evaluation protocol has a load-bearing flaw: the reported ppref values are the maximum over training epochs selected directly on the test set, which inflates the measured performance and can bias the comparison between methods with different training dynamics. The paper also evaluates only on a metric that aligns with the CCR training objective. These issues must be addressed before the quantitative conclusions can be considered reliable.","major_comments":[{"comment":"The reported ppref values are computed as the maximum over all 30 training epochs, evaluated on the same test set used for the final assessment, for each of the five random initializations (Section VI-A: 'The training was run for 30 epochs'; Section VI-B: 'maximum pprefs over all epochs'). No validation split or early-stopping rule is described. This test-set-guided epoch selection biases the reported numbers upward, and because RankNet (CCR) and MSE (ACR) training converge at different rates and have different epochwise variance, the max-over-epoch comparison can systematically favor the method whose epochwise curve happens to peak higher even if the methods perform equally at a principled stopping point. The claim that 'CCR training is superior to ACR training' is therefore not supported by the current protocol. Please repeat the comparison using model selection on a validation split (e.g., select the epoch with best validation ppref) or report per-epoch curves and a comparison at a fixed training duration.","section":"VI-A, VI-B, Figure 4"},{"comment":"The only evaluation metric, ppref, is computed from CCR labels (Section IV-A), which are exactly the labels used in the CCR training objective (Equation 1). ACR training, in contrast, minimizes MSE on five-point ACR scores. Evaluating ACR-trained models on ppref conflates 'better at predicting CCR preferences' with 'better assessment model,' giving CCR training a built-in advantage. The paper should include an ACR-based evaluation metric (e.g., MSE or rank correlation on held-out ACR labels) or otherwise justify that ppref is the appropriate single metric for comparing the two training approaches.","section":"IV-A, V-B"},{"comment":"The SVDs used in the experiments were deliberately pre-selected for low inter-rater variance and relatively large pseudo F values (Section III: 'We deliberately selected SVDs that showed relatively little variation in opinion among participants'). The upper-bound estimates in Table 1 are derived from the same annotator population. This selection biases the task toward easier SVDs, so the claim that 'moderate ppref is achievable with very limited data' may not generalize to SVDs with higher inter-rater disagreement. Please discuss this selection bias and, if possible, report results on at least one SVD with higher variance to assess the robustness of the findings.","section":"III"},{"comment":"The claim that 'CCR training is superior to ACR training' is based on only five random initializations, with no significance testing reported. In Figure 4, the standard deviations appear to overlap for several training-data conditions (e.g., fixedSSL-FC at 5000 samples), so the observed differences may not be statistically reliable. Please report paired significance tests (e.g., Wilcoxon signed-rank test across the five trials) or confidence intervals for the difference in ppref between the two training methods at each training-data size.","section":"VI-B"}],"minor_comments":[{"comment":"There is a typo: 'abosolute' should be 'absolute.'","section":"Abstract"},{"comment":"The BCE loss equation is missing a negative sign and a sum over pairs: it should be L = -Σ[ P_ij log(P̂_ij) + (1-P_ij) log(1-P̂_ij) ]. Please correct the formula.","section":"V-B"},{"comment":"The capitalization of 'ppref' is inconsistent (e.g., 'Ppref' in Table 1 header and 'ppref' in the text). Please standardize.","section":"IV-A"},{"comment":"The phrase 'preliminary casual experiment' is vague; please provide a brief description of how the pseudo F values were computed and how the selection threshold was determined.","section":"III"},{"comment":"The description of the test-set construction is terse: 'the test data are a portion of CCRs assigned to the speaker pair including at least one speaker not included in Strain.' Please clarify how the test pairs were sampled and whether any speakers overlap between training and test pairs.","section":"VI-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the proposed task is of interest, but the evaluation protocol is a serious flaw that undermines the main empirical claims. The authors should be asked to rerun the experiments with a validation-based model-selection procedure and to provide significance testing. Additionally, the novelty of personalized SVD assessment should be discussed in relation to existing personalized MOS prediction work (e.g., LDNet and personalized audio quality preference prediction). If the dataset is not released, the reproducibility of the experimental results is limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper introduces a genuinely new task and a new dataset for subjective voice descriptor (SVD) scoring, and the claim that pairwise comparison (CCR) training beats absolute rating (ACR) training is interesting but not yet convincing because the evaluation protocol selects the best epoch on the test set.\n\nWhat is new: The SVD task framing—training models to score speech on subjective descriptors like 'cute voice' with personalization in mind—is a useful extension of MOS prediction. The dataset construction is thoughtful: they use CCR labels for ease of annotation, include many speakers, and explicitly aim for ranking rather than absolute score accuracy. The upper-bound estimation for ppref from a common questionnaire is a good idea. The finding that limited CCR data can give moderate ppref is plausible and worth exploring.\n\nThe soft spot is real and load-bearing. Figure 4 reports the maximum ppref over all 30 training epochs, selected directly on the test set, for each of five initializations, with no validation split. That is a form of test-set peeking. Taking the per-run maximum over epochs biases the reported number upward, and the bias can differ between methods—RankNet and MSE converge at different rates and have different epochwise variance. The stress-test note is right: the CCR-over-ACR gap could shrink or even reverse under a principled early-stopping rule. This is not a minor detail; it is the paper's central empirical claim.\n\nA secondary concern is that ppref is computed from CCR labels, which is exactly what RankNet optimizes. This is defensible if the task is ranking, but it puts ACR training at a disadvantage. Reporting a metric that does not directly match the ACR objective, such as Spearman correlation on held-out absolute scores, would strengthen the comparison. The pre-selection of SVDs for low inter-rater variance also makes the dataset easier than a real deployment scenario, and the dataset/code are not public, so independent verification is impossible.\n\nWho should read this: people working on speech quality assessment, preference learning for audio, or TTS personalization. The task framing alone justifies a look. But the empirical claim needs a validation-based rerun before I'd trust it.\n\nRecommendation: send to peer review—a solid referee can push for the right fix—but expect major revision on the evaluation protocol.","headline":"New task and dataset for subjective voice descriptor scoring, but the headline CCR-beats-ACR result rests on test-set-guided epoch selection and needs a validation-based check.","tokens_in":8714,"tokens_out":2622,"would_cite":true,"duration_ms":30126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that pairwise comparison labels train a voice-assessing model better than absolute ratings, even with very little data.","keywords":["subjective voice descriptors","automatic speech assessment","comparison category rating","absolute category rating","RankNet","precision of preferences","personalized voice scoring","preference learning"],"falsifier":"Run the same experiment with early stopping on a held-out validation set rather than the test set; if ACR training then matches or exceeds CCR training in ordering accuracy, the paper's main claim is falsified. Alternatively, on a subjective attribute where absolute ratings are known to be highly reliable and pairwise comparisons are noisy, ACR training would be expected to win, which would refute the claimed universal superiority of CCR training.","tokens_in":7686,"feed_emoji":"🗣️","tokens_out":4807,"duration_ms":49344,"temperature":0.7,"pith_summary":"The authors introduce a new task: training neural networks to rate speech according to subjective descriptors such as 'youthful-sounding' or 'resonant' voice. They compare two training regimes: regression on absolute category ratings (ACR, a 1-to-5 scale) and RankNet learning on comparison category ratings (CCR, where a listener says which of two voices is more 'X'). Using a custom speech dataset with both label types, they evaluate ordering accuracy with ppref, the precision of predicted score orderings on held-out comparison pairs. Across three descriptors and two model architectures, CCR-trained models reach moderate ordering accuracy even with only 125 to 500 training labels, and CCR training consistently beats ACR training at every data size. The conclusion is that pairwise preference labels are a reliable, label-efficient route for learning personalized subjective voice assessments.","feed_headline":"Pairwise comparisons beat 1-to-5 ratings for voice scoring","feed_subtitle":"A small set of 'which voice is more X' labels outperforms absolute scores across all data sizes.","key_machinery":"The central mechanism is RankNet applied to CCR labels: a network takes two speech representations, produces scores for each, and is trained with binary cross-entropy on the sigmoid of their score difference, with the four CCR response options mapped to target probabilities 0, 0.25, 0.75, and 1. The evaluation metric ppref (precision of preferences) is the accuracy of the predicted score ordering on held-out CCR pairs, with separate strong and weak variants. This machinery carries the argument because it lets a model learn a scalar score from pairwise comparisons alone, and the metric directly measures whether the learned score reproduces human preference orderings.","core_discovery":"The central discovery is that comparison category ratings, though collected on a small random subset of all possible pairs, teach a score-predicting network to order voices by a subjective descriptor more accurately than an equal-sized set of absolute ratings. This holds for three descriptors (youthful female voice, youthful male voice, resonant male voice) and two architectures: a MOSNet-style CNN-BLSTM-FC and a fixed wav2vec2.0 feature extractor with a small trainable head. The authors attribute the advantage to the higher reliability of pairwise comparison labels, since listeners agree more readily on which of two voices is 'more' something than on assigning an absolute number. They also find that ppref-strong, computed on confident preference labels, improves with more training data and exceeds ppref-weak, computed on 'a little more' labels, suggesting that low-confidence comparisons are noisy and should be filtered out when evaluating performance.","pith_inferences":["The paper selects, for each random initialization, the training epoch with the highest test-set ordering accuracy; an independent replication with validation-based early stopping would reveal how much of the CCR-over-ACR advantage survives without test-set peeking.","The same CCR-versus-ACR comparison could be tested in other subjective domains, such as image aesthetics or product review helpfulness, where pairwise preferences are also easier to collect than absolute scores.","Because CCR labels appear more reliable, a multi-task loss combining CCR and ACR labels might push ordering accuracy beyond CCR alone, a combination the paper does not test.","The weak 'a little more' labels contributed little signal in this study, suggesting that a forced two-option choice without intensity distinctions could be a more label-efficient collection strategy for personalized models."],"forward_implications":["A single user's pairwise 'which is more X' judgments, even numbering in the hundreds, can ground a usable voice-impression scorer for that user.","For subjective voice descriptor assessment, CCR-based training should be preferred over ACR-based regression, especially in the low-data regime where personalized models operate.","The ppref-strong metric is a suitable primary evaluation for such models, and weak 'a little more' comparisons should be discarded or down-weighted in evaluation.","The data-collection protocol and training method extend beyond fixed descriptors to personalized descriptors like 'my favorite voice' because they tolerate very small label budgets."],"supporting_citations":[{"why":"Supplies the MOSNet CNN-BLSTM-FC architecture used as one of the assessment models.","marker":"[2]"},{"why":"Defines the ACR and CCR category rating schemes used for data collection and training labels.","marker":"[6]"},{"why":"Introduces the precision-of-preferences (ppref) metric and the idea of evaluating ranking by pairwise preference accuracy.","marker":"[7]"},{"why":"Provides evidence that forced-choice comparison labeling is more accurate and faster than rating, supporting the CCR design.","marker":"[8]"},{"why":"Provides the wav2vec2.0 self-supervised speech features used by the fixedSSL-FC model.","marker":"[21]"},{"why":"Supplies the RankNet training loss and update procedure used for CCR-based training.","marker":"[24]"}],"fun_headline_variants":["Pairwise comparisons beat absolute ratings for voice scoring","Voice impression models learn better from pairwise preference data","For subjective voice descriptors, comparisons beat absolute scores","RankNet on pairwise comparisons beats regression on absolute voice scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline comparison comes from selecting, for each random initialization, the training epoch with the best test-set ordering accuracy, so the reported advantage of CCR training over ACR training could depend on test-set information leaking into model selection.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise comparisons beat absolute ratings for voice scoring","Voice impression models learn better from pairwise preference data","For subjective voice descriptors, comparisons beat absolute scores","RankNet on pairwise comparisons beats regression on absolute voice scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3626,"prompt_tokens":943,"completion_tokens":2683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2622}},"tokens_in":559,"tokens_out":2683,"duration_ms":20237,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:20.192113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiment with early stopping on a held-out validation set rather than the test set; if ACR training then matches or exceeds CCR training in ordering accuracy, the paper's main claim is falsified. Alternatively, on a subjective attribute where absolute ratings are known to be highly reliable and pairwise comparisons are noisy, ACR training would be expected to win, which would refute the claimed universal superiority of CCR training.","supporting_citations":[{"cited_title":"MOSNet: Deep learning based objective assessment for voice conversion,","cited_arxiv_id":null,"evidence_quote":"Supplies the MOSNet CNN-BLSTM-FC architecture used as one of the assessment models."},{"cited_title":"Methods for subjective determination of transmis- sion quality,","cited_arxiv_id":null,"evidence_quote":"Defines the ACR and CCR category rating schemes used for data collection and training labels."},{"cited_title":"Here or There: Preference Judgments for Relevance,","cited_arxiv_id":null,"evidence_quote":"Introduces the precision-of-preferences (ppref) metric and the idea of evaluating ranking by pairwise preference accuracy."},{"cited_title":"Comparison of four subjective methods for image quality assessment,","cited_arxiv_id":null,"evidence_quote":"Provides evidence that forced-choice comparison labeling is more accurate and faster than rating, supporting the CCR design."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"Provides the wav2vec2.0 self-supervised speech features used by the fixedSSL-FC model."},{"cited_title":"Learning to Rank Using Gradient Descent,","cited_arxiv_id":null,"evidence_quote":"Supplies the RankNet training loss and update procedure used for CCR-based training."}],"review_version":1}