{"id":"da4faa99-ddfa-4a54-b904-69e45d5da0e6","arxiv_id":"2506.02232","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Speaker-recognition pre-trained models (x-vector, ECAPA) outperform other speech and music models for singing voice MOS prediction, and their fusion via a Bhattacharyya-distance loss sets a new reported state of the art on SingMOS.","lead":"A team of researchers tested whether speaker recognition models can judge singing voice quality, and found that x-vector and ECAPA features work best for predicting human quality scores. They also introduced a fusion method called BATCH that combines features from two models and claims state-of-the-art accuracy on the SingMOS benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BATCH's Bhattacharyya loss is undefined as written: raw gated features are not normalized probability distributions, so the SOTA fusion claim depends on an unverified objective.","rationale":"Good-faith reading: the paper's core empirical observation is that speaker-recognition SPTMs (x-vector, ECAPA) outperform other PTMs on SingMOS, and that fusing them helps. That observation may well be true and is worth reporting. However, the paper's headline contribution is BATCH, and Table 2's dramatic gains (test-other1 MAE 0.14 vs 0.71 for the best individual model) are attributed to BATCH. The description of L_BD in Section 3.2 is internally incomplete: Bhattacharyya distance requires normalized, nonnegative distributions, and no such normalization or nonnegativity is specified. The text's 'position x' sum reads as a sum over vector elements of raw gated features, which can be negative (the sigmoid gate preserves sign) and are not unit-sum. This is not a matter of tuning; it determines whether the loss is computable and whether minimizing it encourages alignment or magnitude growth. The reader's overfitting concern is also real and important, but it is downstream of this issue: if the objective is unspecified, no amount of seeding or error bars makes the SOTA claim reproducible. The most decisive test is to inspect the released code or re-implement the loss as a true normalized Bhattacharyya distance and check whether the reported XV+EC numbers survive. If the code reveals the intended normalization, this critique dissolves; if not, the central claim should be revised to remove the SOTA statement until the fusion method is properly specified. Because the reader already recommends REJECT, my verdict remains unchanged.","tokens_in":11199,"tokens_out":5977,"duration_ms":54910,"concrete_test":"Run the official BATCH code from https://github.com/Helix-IIIT-Delhi/BATCH-SingMOS (or re-implement from Section 3.2) on the SingMOS train split with the Bhattacharyya coefficient computed on gated features normalized by their L1 norm, so that sum_x sqrt(P(x)Q(x)) is a true coefficient in [0,1], and evaluate XV+EC on test-other1 and test-main. If the MAE does not reproduce Table 2's 0.14/0.09 values, the reported fusion gains come from a different, undocumented objective. Additionally, compare this corrected run to concatenation fusion with identical seeds and early stopping to check whether the fusion benefit survives a properly defined Bhattacharyya loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines BATCH's loss as L_total = L_MSE + alpha * L_BD with L_BD = -log sum_x sqrt(P(x)Q(x)), where P(x) and Q(x) are called feature distributions of the two PTMs at position x. For a valid Bhattacharyya distance, P and Q must be nonnegative and normalized probability mass functions over the same index set. The paper never states that the gated feature maps are normalized to sum to 1, nor does it establish nonnegativity: the gating G(x)=sigma(x)⊙x preserves the sign of x, and no ReLU or softmax is specified for the features before the sqrt(P(x)Q(x)) computation. If any product P(x)Q(x) is negative, the square root is undefined in the real numbers; if values are nonnegative but unnormalized, the sum can exceed 1, making -log(sum) negative, so minimizing L_BD would drive feature magnitudes upward rather than aligning distributions. Either way, the described objective cannot be verified as the one that produced the reported numbers. Because BATCH is the paper's novelty and the SOTA claim is specifically 'through BATCH with the fusion of x-vector and ECAPA', this ambiguity is load-bearing. The Table 2 ablation against concatenation also loses interpretability: the baseline removes the gate and the BD term, so any gain could come from the gating mechanism, the extra loss, or both. Until the code is released or the paper specifies a normalized, nonnegative construction, the central claim is not reproducible from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates thirteen pre-trained models for SingMOS prediction, including speaker-recognition embeddings (x-vector, ECAPA), self-supervised speech models (WavLM, Wav2vec2, Unispeech-SAT, XLS-R, Whisper, MMS), and music models (music2vec, MERT variants), using FCN and CNN heads. It reports that x-vector and ECAPA give the best individual results and that a new fusion method, BATCH, combining gated features with a Bhattacharyya-distance loss, achieves the best overall scores, claimed to be state of the art. The experiments use the SingMOS benchmark and compare with concatenation fusion.","tokens_in":11538,"tokens_out":9200,"duration_ms":81975,"significance":"The paper addresses an under-explored task and provides a broad, systematic comparison using public checkpoints and a public benchmark. If the findings are robust, the conclusion that speaker-recognition embeddings are strong features for singing-voice MOS is practically useful and could reorient feature selection for singing quality assessment. The authors are appropriately explicit that this is a hypothesis rather than a proven causal mechanism. However, the central BATCH contribution is not reproducible as written, and the SOTA claim lacks direct comparison; the paper is not yet ready to support its strongest conclusions.","major_comments":[{"comment":"The Bhattacharyya loss is undefined for the features produced by the described architecture. The gated features G(x)=σ(x)⊙x preserve the sign of x, and no non-negativity or normalization step is specified; therefore P(x) and Q(x) can take negative values, making sqrt(P(x)Q(x)) non-real, and if they are nonnegative but unnormalized, Σ_x sqrt(P(x)Q(x)) is not bounded by 1, so minimizing -log of that sum does not correspond to distribution alignment. Please specify a valid construction (e.g., softmax/probability normalization with a non-negativity guarantee) and verify that the reported numbers are produced by that construction; otherwise the BATCH results cannot be reproduced from the manuscript.","section":"3.2"},{"comment":"The claim of state-of-the-art performance is not supported by any direct comparison. The paragraph 'Comparison to Methods used in Previous Works' names Tang et al. [9] and [10], but Table 2 reports only this paper's systems. Add a table with the published SingMOS MAE/MSE values for the prior methods on the same test splits, and state the selection criterion (e.g., best MAE or MSE) for the SOTA claim.","section":"4.2"},{"comment":"All results are single-run with no seed variance or confidence intervals. The SingMOS training split is small (3,421 clips total, Section 4.1), while BATCH has 2M–6M trainable parameters, so the dramatic test-other1 CNN improvement from MAE 0.71 (x-vector) to 0.14 (x-vector+ECAPA) could reflect overfitting, early-stopping choices, or lucky initialization rather than generalization. Report results over multiple seeds with mean±std and describe the early-stopping and α-selection protocol; otherwise the magnitude of the fusion gain is not interpretable.","section":"4.1/4.2"},{"comment":"The concatenation baseline removes both the gating mechanism and the BD loss, so the reported BATCH-vs-concatenation gains cannot be attributed to the Bhattacharyya loss, the gating, or their interaction. Add ablations that include gate-only and BD-only variants (and, if applicable, the normalization step) to isolate the contribution of the proposed loss.","section":"3.2/Table 2"}],"minor_comments":[{"comment":"Table 1 lists 'Hu (HuBERT)' in the abbreviations but contains no HuBERT column; either include the HuBERT results or remove the mention in Section 2.","section":"Table 1"},{"comment":"The parameter counts '94.70M, 94.68M, and 95.04M' are given for WavLM, Unispeech-SAT, HuBERT, and Wav2vec2, but only three numbers are provided for four models.","section":"Section 2"},{"comment":"Table 2 contains two rows labeled 'MT95+MTV0' with different scores; one label is likely a typo and should be corrected.","section":"Table 2"},{"comment":"The paper uses 'Bhattacharya' in several places; the conventional spelling is 'Bhattacharyya'.","section":"Throughout"},{"comment":"There are minor language issues, including 'in a much more better way' in the abstract and 'continous' in Section 3.2; these should be corrected.","section":"Abstract/3.2"},{"comment":"MAE/MSE values are sometimes written with a '%' sign (e.g., '0.91%'); these are absolute errors, not percentages, and should be presented without the percent sign.","section":"Section 4.2/Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the reported BATCH numbers may not correspond to the loss as written. Because no code or seeds are provided, I could not verify the core experiment. I would advise the editor that acceptance should require code release and a corrected, reproducible formulation; otherwise the SOTA claim should be withdrawn."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper before deciding how much time to spend on it. The empirical comparison of individual pre-trained models is genuinely new and probably right: x-vector and ECAPA, both speaker-recognition embeddings, clearly beat the speech and music PTMs on both SingMOS test sets. That is a useful result for anyone building singing quality predictors. The second thing is that the paper's headline contribution—BATCH fusion with a Bhattacharyya-distance loss—is not mathematically well-defined as written, and the reported fusion gains are suspiciously large.\n\nWhat the paper does well: it evaluates a wide range of PTMs (monolingual, multilingual, speaker-recognition, music) on the SingMOS benchmark, with both FCN and CNN downstream heads, and the ranking is consistent across test sets. The authors also correctly point out that the multilingual PTMs don't automatically win despite being exposed to Chinese and Japanese, which strengthens their claim that speaker-recognition pretraining captures something useful for singing voice.\n\nThe soft spots are real. First, the BD loss: L_BD = -log(sum sqrt(P(x)Q(x))) requires P and Q to be nonnegative and sum to 1. The paper never normalizes the gated features, and the gating G(x)=sigmoid(x)*x preserves the sign of x. If any product is negative, the square root is undefined; if all are positive but unnormalized, minimizing -log(sum) pushes magnitudes up, not toward alignment. As written, the objective cannot be the one that produced the numbers. This is load-bearing because the SOTA claim is specifically about BATCH.\n\nSecond, there is no comparison to the published SOTA numbers from Tang et al. or the ISCSLP paper. The authors say \"setting SOTA\" but only compare against their own re-runs, which is not the same as beating the best published result. Third, no error bars or seed variance on a dataset of only ~3,400 clips; the MAE drops from 0.71 to 0.14 on test-other1 are large enough to make me wonder about overfitting or evaluation artifacts. Fourth, the ablation in Table 2 removes both the gate and the BD term together, so you cannot tell which component causes the gain.\n\nThese issues are fixable. A revision with normalized features, a direct SOTA comparison, error bars, and a proper component ablation could turn this into a solid contribution. As it stands, the individual PTM finding is worth citing, but the BATCH claims need verification.\n\nMy recommendation: send it to peer review, not desk reject. The first-result on speaker-recognition PTMs is valuable enough that a careful referee can help the authors fix the technical flaws. I would bring it to a reading group only after they address the BD definition.","headline":"Useful empirical finding on speaker-recognition features for SingMOS, but the BATCH fusion loss is under-specified and the SOTA claim outruns the evidence.","tokens_in":12057,"tokens_out":3248,"would_cite":true,"duration_ms":28490,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speaker-recognition pre-trained models are the most effective features for singing-voice quality prediction, and fusing x-vector with ECAPA via the BATCH framework sets a new state of the art on the SingMOS benchmark.","keywords":["SingMOS prediction","singing voice quality assessment","speaker recognition pre-trained models","x-vector","ECAPA","Bhattacharyya distance fusion","pre-trained model fusion","self-supervised speech representations"],"falsifier":"Retrain the best BATCH model (x-vector + ECAPA, CNN head) on the SingMOS train split with shuffled MOS labels: if it still achieves test-other1 MAE near 0.14, the loss is not learning quality; alternatively, run the reported 0.14-MAE configuration over 10 random seeds and require the mean and worst-case MAE to stay far below the 0.71 single-model baseline before accepting the gain as generalization.","tokens_in":11030,"feed_emoji":"🎤","tokens_out":6036,"duration_ms":50224,"temperature":0.7,"pith_summary":"The paper claims that pre-trained speaker-recognition models—x-vector and ECAPA—extract the features most useful for predicting how human listeners rate synthesized singing voices, outperforming general speech self-supervised models and music pre-trained models. It tests this on the SingMOS benchmark with both fully connected and convolutional downstream heads, and the speaker-recognition models give the lowest errors on both test splits. The paper then introduces BATCH, a fusion method that aligns two pre-trained models' feature distributions with a Bhattacharyya-distance loss before concatenating them. Fusing x-vector and ECAPA with BATCH produces the best reported SingMOS scores, improving the CNN test-other1 MAE from 0.71 for the best single model to 0.14. If the result holds, singer-identity-oriented pretraining is a better source of fine-grained vocal cues for quality assessment than the usual speech- or music-representation targets.","feed_headline":"Two speaker-ID models, fused, set SingMOS state of the art","feed_subtitle":"x-vector and ECAPA, aligned with a Bhattacharyya-distance loss, outperform speech and music pretrained models on singing-voice quality.","key_machinery":"The load-bearing mechanism is BATCH (fusion via Bhattacharyya distance), which combines two frozen pre-trained representations through a gating operation $G(x)=\\sigma(x)\\odot x$, then aligns the two gated feature distributions with the Bhattacharyya distance $D_B(P,Q)=-\\log\\sum_x\\sqrt{P(x)Q(x)}$ added as a loss term to mean squared error: $L = L_{\\mathrm{MSE}} + \\alpha L_{BD}$ with $\\alpha=0.3$. The Bhattacharyya distance is the workhorse: unlike cosine similarity or KL divergence, it compares corresponding elements pointwise and is claimed to align heterogeneous feature spaces more stably, penalizing distant representations while encouraging compactness. After alignment the gated features are concatenated and fed to a 128-neuron fully connected regressor. BATCH adds only 2M-6M trainable parameters, so the gains come from the alignment loss, not from a large head.","core_discovery":"On the paper's own terms, the central discovery is that models trained to recognize who is speaking generalize to judging how well someone sings. x-vector and ECAPA, both time-delay neural networks trained on VoxCeleb, consistently outperform monolingual, multilingual, and music PTMs with both FCN and CNN heads. The authors attribute this to speaker-recognition pretraining encoding pitch, tone, intensity, and rhythm—the same cues listeners use to judge synthesized singing quality. The second discovery is that the two speaker models are complementary: aligning their representations with the BATCH loss and concatenating them yields a CNN test-other1 MAE of 0.14, far below the best individual model's 0.71, and likewise tops test-main. The paper reports that this combination exceeds previous SingMOS methods.","pith_inferences":["The magnitude of the reported gain (test-other1 CNN MAE 0.71 to 0.14) is unusually large for a small training set; one plausible explanation the paper does not test is that the Bhattacharyya loss acts as a strong regularizer on the 2M-6M trainable parameters, and ablating $\\alpha$ would reveal how much of the gain is alignment versus regularization.","An extension the paper leaves implicit is applying BATCH to speech MOS or to non-intrusive assessment of other generative audio, where two complementary pretrained encoders could be aligned the same way.","The claim that speaker-identity pretraining is the key mechanism is inferred from model choice, not directly probed; one could test it by fine-tuning x-vector on a non-speaker task and checking whether the SingMOS advantage disappears."],"forward_implications":["Future SingMOS systems should include speaker-recognition embeddings such as x-vector and ECAPA as default baselines, since they beat both self-supervised speech models and music models in this study.","Fusing complementary pre-trained models with a distribution-alignment loss is a viable alternative to simple concatenation; BATCH beats concatenation across most of the 78 combinations tested.","The two speaker-recognition models encode complementary information, so their combination is not redundant; the paper reports this as the top-performing fusion.","Because the gains appear with both FCN and CNN heads on both test splits, the finding is not tied to one downstream architecture."],"supporting_citations":[{"why":"Provides the SingMOS dataset and evaluation protocol used for all experiments.","marker":"[9]"},{"why":"Supplies the x-vector speaker-recognition embeddings that, fused with ECAPA, set the state of the art.","marker":"[15]"},{"why":"Supplies the ECAPA speaker-recognition embeddings whose fusion with x-vector gives the best reported scores.","marker":"[16]"},{"why":"Establishes the previous SingMOS prediction approaches and baselines that this work extends and surpasses.","marker":"[10]"},{"why":"Motivates fusing pretrained models for MOS prediction, which BATCH operationalizes with a Bhattacharyya-distance loss.","marker":"[12]"},{"why":"Provides the MERT music pre-trained models used as music-domain baselines in the comparison.","marker":"[24]"}],"fun_headline_variants":["Speaker-ID models beat music PTMs for SingMOS prediction","Fusing x-vector and ECAPA sets SingMOS SOTA","Speaker recognition pretraining transfers to singing quality","BATCH fusion of speaker models tops SingMOS benchmarks","Why speaker-ID models excel at SingMOS: fusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that BATCH's 2M-6M trainable parameters generalize from the few thousand SingMOS training clips rather than memorizing them; if the dramatic test-set drops reflect overfitting or accidental data-selection, the state-of-the-art claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Speaker-ID models beat music PTMs for SingMOS prediction","Fusing x-vector and ECAPA sets SingMOS SOTA","Speaker recognition pretraining transfers to singing quality","BATCH fusion of speaker models tops SingMOS benchmarks","Why speaker-ID models excel at SingMOS: fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1743,"prompt_tokens":907,"completion_tokens":836,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":758}},"tokens_in":523,"tokens_out":836,"duration_ms":7024,"temperature":1.0,"reasoning_tokens":758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:27:20.007508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best BATCH model (x-vector + ECAPA, CNN head) on the SingMOS train split with shuffled MOS labels: if it still achieves test-other1 MAE near 0.14, the loss is not learning quality; alternatively, run the reported 0.14-MAE configuration over 10 random seeds and require the mean and worst-case MAE to stay far below the 0.71 single-model baseline before accepting the gain as generalization.","supporting_citations":[{"cited_title":"Investigating the Reasonable Effectiveness of Speaker Pre-Trained Models and their Synergistic Power for SingMOS Prediction","cited_arxiv_id":"2506.02232","evidence_quote":"Provides the SingMOS dataset and evaluation protocol used for all experiments."},{"cited_title":"An exploration on singing mos prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the ECAPA speaker-recognition embeddings whose fusion with x-vector gives the best reported scores."},{"cited_title":"Le- ssl-mos: Self-supervised learning mos prediction with listener enhancement,","cited_arxiv_id":null,"evidence_quote":"Establishes the previous SingMOS prediction approaches and baselines that this work extends and surpasses."},{"cited_title":"Ensemble of deep neural network models for mos prediction,","cited_arxiv_id":null,"evidence_quote":"Motivates fusing pretrained models for MOS prediction, which BATCH operationalizes with a Bhattacharyya-distance loss."},{"cited_title":"Unispeech-sat: Universal speech repre- sentation learning with speaker aware pre-training,","cited_arxiv_id":null,"evidence_quote":"Provides the MERT music pre-trained models used as music-domain baselines in the comparison."}],"review_version":1}