{"id":"05a85634-6b01-4c3e-915c-37ce4a507549","arxiv_id":"2502.04722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using WavLM self-supervised features with fine-tuning and weighted layer summation, the paper reports improved melody preservation in singing voice conversion with background music compared to PYIN and Crepe baselines.","lead":"This paper introduces a singing voice conversion system that uses features from self-supervised speech models to preserve the melody even when the input song contains background music. It reports better pitch accuracy and naturalness than current methods, particularly in noisy conditions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Objective melody metrics are partly circular: the melody extractor is trained on the same median PYIN/REAPER/Parselmouth pitch labels used to compute F0RMSE/F0CORR, so the reported advantage may reflect tracker affinity rather than true melody preservation.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the objective metrics are partly self-referential because the melody extractor is trained and evaluated on the same family of pitch estimators. This is not a minor methodological nit; it directly undermines the 'melody accuracy' half of the abstract's central claim. The absence of an independent ground truth means Table I and Table II cannot currently distinguish 'better melody preservation' from 'better prediction of median-PYIN/REAPER/Parselmouth outputs.' The paper does have genuine strengths: a clear framework, ablations of SSL layers and fine-tuning, and subjective MOS results that, while limited (15 volunteers, no significance testing), show a plausible direction. However, the objective numbers are too close to the Separated+Crepe baseline in noisy conditions to support 'significantly outperforms' without an independent metric. The proposed concrete test would settle the question because it removes the circularity: if manual or otherwise independent pitch annotation reproduces the reported ranking, the concern is resolved; if not, the objective claim fails. Since the reader's verdict is already CONDITIONAL and this test is exactly the kind of condition that should be imposed, no verdict change is needed.","tokens_in":8726,"tokens_out":3682,"duration_ms":41507,"concrete_test":"Recompute F0RMSE and F0CORR on the same test set using an independent, perceptually validated pitch reference: have trained musicians annotate note-by-note pitch on a subset of the source and converted samples (or use a high-accuracy offline pitch estimator not in the PYIN/REAPER/Parselmouth family), then compare proposed versus Separated+Crepe and Crepe on this reference. If the proposed advantage disappears, shrinks to statistical insignificance, or reverses, the melody-accuracy claim is a tracker-affinity artifact rather than a genuine improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed WavLM-based melody features significantly improve melody accuracy in noisy and clean conditions. Section III-B states that the melody extractor is trained with ground-truth pitches 'extracted from clean audio by taking the median of PYIN, REAPER, and Parselmouth.' Section IV-A then evaluates F0RMSE and F0CORR using F0 contours extracted with the same family of trackers. This is a circular evaluation: the model is scored on how closely it reproduces the behavior of the very estimators it was trained to mimic. The 'Original Pitch&Energy' baseline in Table II uses exactly that same median estimator as input, so the clean-condition comparison is especially confounded: the proposed model is trained to predict those labels, while Crepe and Separated+Crepe are not. At 0 dB SNR, the proposed method's advantage over Separated+Crepe is tiny (F0RMSE 0.199 vs 0.202; F0CORR 0.935 vs 0.928), and on clean audio proposed is slightly worse than Crepe on both F0RMSE (0.169 vs 0.164) and F0CORR (0.960 vs 0.967). With no significance testing and no independent perceptual ground truth, the objective 'melody accuracy' component of the central claim rests on a metric that may systematically favor the proposed extractor without reflecting genuinely better melody preservation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an any-to-one singing voice conversion (SVC) system that uses self-supervised learning (SSL) representations, specifically WavLM and HuBERT, to extract melody features that are robust to background music (BGM). The melody extractor combines weighted-summed SSL hidden states with FFT blocks, and is fine-tuned on BGM-mixed data. The proposed system is compared against three baselines (Original Pitch&Energy, Crepe, Separated+Crepe) under clean and noisy (0, 5, 10, 15 dB SNR) conditions, using F0RMSE, F0CORR, and MOS. The central claim is that the proposed method significantly outperforms baselines in melody accuracy and in subjective similarity/naturalness, especially at low SNR.","tokens_in":9064,"tokens_out":7075,"duration_ms":65278,"significance":"If the claimed improvements are substantiated, the paper would offer a practical solution for SVC with accompaniment, which is a real and underexplored problem. The layer-weight analysis (Fig. 3) and the comparison of SSL variants are also useful empirical contributions. However, the current evaluation has several confounds: the objective pitch metrics may favor the proposed extractor due to being trained on the same pitch-estimator family, no significance tests support the 'significantly outperforms' wording, the clean-condition MOS advantage is within overlapping confidence intervals, and the Crepe baseline is not adapted to BGM data. These issues currently prevent the evidence from carrying the strength of the central claim.","major_comments":[{"comment":"The melody extractor is trained with ground-truth pitch labels defined as the median of PYIN, REAPER, and Parselmouth (Section III-B). The objective evaluation in Section IV-A computes F0RMSE and F0CORR between converted and source audio, but the manuscript does not state which F0 extraction method is used for these metrics. If the same PYIN/REAPER/Parselmouth family is used, the proposed model is being evaluated on how well it reproduces the behavior of the very estimators it was trained to imitate, while the Original Pitch&Energy baseline consumes those same median labels without a learned module trained to reproduce them. Please specify the F0 extraction method used in the evaluation, and if it coincides with the training labels, add an evaluation against an independent pitch reference (e.g., a different pitch tracker or manual annotations) and include a control where Crepe is also fine-tuned on the BGM-mixed training set.","section":"Section III-B and IV-A"},{"comment":"The abstract and Section V claim that the proposed method 'significantly outperforms' baselines, but no significance tests are reported for F0RMSE or F0CORR, and the MOS 95% confidence intervals on clean audio overlap with the Original Pitch&Energy baseline (NMOS 3.54±0.14 vs 3.38±0.14; SMOS 3.71±0.14 vs 3.58±0.13). At 0 dB SNR, the objective gain over Separated+Crepe is small (F0RMSE 0.199 vs 0.202; F0CORR 0.935 vs 0.928). Please report paired significance tests across test items and listeners, or temper the 'significantly' language to match the actual strength of the evidence.","section":"Table II and Abstract"},{"comment":"The proposed melody extractor is trained on BGM-accompanied singing data and its SSL model is fine-tuned, whereas the Crepe baseline is used as a pretrained model without any adaptation to BGM. The observed advantage may therefore be due to the additional BGM training data rather than to the SSL representation per se. To isolate the contribution of SSL-based features, please add a baseline where Crepe is fine-tuned on the same BGM data, or an ablation that uses the same architecture and training data but a non-SSL front-end.","section":"Section III-B and IV-B"},{"comment":"The manuscript describes RobustSVC (Ref. [22]) as the backbone of the proposed framework, but RobustSVC is not included in the comparison in Table II. Since RobustSVC is a directly comparable SVC method for the same noisy-condition setting, its omission makes the 'comparison with state-of-the-art methods' incomplete. Please add RobustSVC as a baseline or provide a clear justification for its exclusion.","section":"Section IV-B and Ref [22]"}],"minor_comments":[{"comment":"The definition of F0RMSE is confusing: 'The RMSE between the converted and source waveforms is computed in the absence of BGM' should refer to F0 contours, not waveforms, and 'source' needs to be defined (original clean vocal? original mixture?). Also, F0RMSE is an accuracy metric and should not be described as measuring naturalness.","section":"Section IV-A"},{"comment":"The table header 'Method Condition' is odd because the Condition column is empty, and the table does not indicate which SNR condition the results correspond to. Please clarify whether the numbers are averaged over SNR levels and what the checkmarks in the FT/WS/FFT columns denote.","section":"Table I"},{"comment":"The sentence 'we fine-tune the pretrained HiFi-GAN vocoder using ground-truth-aligned [26]' is incomplete and Reference [26] does not describe a method called 'ground-truth-aligned.' Please rewrite to describe the fine-tuning procedure and the reference appropriately.","section":"Section II-B (last paragraph)"},{"comment":"The y-axis of Fig. 3 is unlabeled, so it is unclear what the plotted weight values represent. Also, the claim that only shallow layers capture melody information in the pretrained model is based on learned weights, which may not fully reflect the information content; consider supporting it with a layer-wise probe or stating the limitation.","section":"Section IV-C and Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the proposed approach is plausible, but the evaluation does not yet substantiate the 'significantly outperforms' claim. The most serious issue is the potential circularity of the objective pitch metrics, combined with the absence of significance tests and the omission of the RobustSVC baseline. The reported low-SNR gains over Separated+Crepe are small, and the clean-condition MOS advantage is not significant by interval overlap. These are fixable with additional baselines, significance testing, and a clear statement of the evaluation F0 extractor, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a modest but legitimate step forward in singing voice conversion with background music. The core idea—use weighted-sum of WavLM layers, fine-tune briefly, and add FFT blocks to extract melody features—is a natural extension of the authors' RobustSVC [22]. The ablations in Table 1 are useful: they show that weighted-sum and fine-tuning both help, and the layer-weight visualization in Fig. 3 is a nice observation (shallow layers capture melody in pretrained SSL, fine-tuning activates deeper layers).\n\nWhat is genuinely new is the specific combination, plus the comparison of HuBERT and WavLM for this task. The claim to be 'first to investigate how SSL benefits melody extraction' is overblown; the paper does not survey the literature closely enough to back that. But the experimental work on SSL model choice is still valuable.\n\nNow the soft spots. First, the missing RobustSVC baseline is a real problem. Since your backbone is exactly that system, you need to show what the changes buy you against it. The current Table 2 compares against Crepe and Demucs+Crepe, but not against [22]. Second, the metric issue flagged in the stress test is real. The melody extractor is trained to predict the median of PYIN, REAPER, and Parselmouth, and F0RMSE/F0CORR are computed against the same family. The model will naturally get high agreement with those trackers. The fact that the proposed method is slightly worse than Crepe on clean audio (F0RMSE 0.169 vs 0.164) is consistent with this concern. A human-perceptual pitch ground truth or at least an independent tracker family would strengthen the claim. Third, the baselines are not trained on BGM-mixed data, while the proposed extractor is. That is an unfair setup. Fine-tune Crepe on the same conditions, or add a BGM-adapted Crepe baseline. Fourth, there are no significance tests for the objective metrics, and the MOS confidence intervals overlap in clean conditions. 'Significantly outperforms' is supported only at low SNR.\n\nThese are all fixable in revision. The paper is not a broad scientific shift, but for practitioners in SVC it addresses a real pain point (source separation artifacts) with a sensible method. The subjective gains at 0 dB SNR are convincing; the clean-condition claims should be toned down.\n\nMy recommendation: send to peer review, not desk reject. A competent referee can push for the missing baseline, fairer comparisons, and a better metric. It is below the bar for the strongest journals, but fine for a good workshop or conference.","headline":"Useful incremental extension of the authors' own RobustSVC with WavLM and weighted-sum, but the melody metric is partly self-referential and the baselines are not quite fair; worth a serious referee.","tokens_in":9546,"tokens_out":3744,"would_cite":false,"duration_ms":35673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing explicit pitch trackers with learned, weighted sums of self-supervised speech representations (WavLM/HuBERT) as melody features makes any-to-one singing voice conversion accurate under background music…","keywords":["singing voice conversion","self-supervised learning","melody extraction","WavLM","HuBERT","background music robustness","pitch tracking","source separation"],"falsifier":"Take a set of songs with a known reference melody (for example, synthesized or performed against a precise MIDI score), mix in accompaniment at 0 dB SNR, convert with the proposed WavLM system, and compare the converted pitch to the known notes rather than to a tracker-derived contour; if the proposed method does not keep its advantage over Crepe and Separated+Crepe against this reference, the central claim of superior melody accuracy fails.","tokens_in":8568,"feed_emoji":"🎤","tokens_out":9924,"duration_ms":89316,"temperature":0.7,"pith_summary":"Singing voice conversion usually assumes the input is a clean vocal track, so background music makes pitch extractors fail and converted vocals go off-key. This paper tries to fix that by taking melody information not from a dedicated pitch tracker but from the internal layers of self-supervised speech models (HuBERT and WavLM), combining all layers with learned weights and running them through transformer blocks. The claimed result is that this melody feature, especially from WavLM, keeps pitch accurate even at 0 dB signal-to-noise ratio: pitch root-mean-square error falls from 0.297 with Crepe to 0.199, and pitch correlation rises from 0.815 to 0.935. If the claim holds, a singing voice converter can work directly on accompanied audio without a source-separation stage and its artifacts, preserving both melody and target-speaker similarity.","feed_headline":"WavLM melody features keep singing conversion on pitch under loud BGM","feed_subtitle":"At 0 dB SNR, pitch error drops from 0.297 to 0.199 and correlation rises to 0.935, no separation step.","key_machinery":"The central object is the SSL-based melody extractor: a pretrained HuBERT or WavLM model whose per-layer hidden states are combined into a weighted sum, $O_t=\\sum_{l=0}^{L} w_l h_l^t$, with learnable weights $w_l$; the summed representation is processed by Feed Forward Transformer blocks to predict pitch, energy, and voiced/unvoiced flags, and the final 256-dimensional block output becomes the melody feature fed to the SVC encoder. The mechanism that carries the argument is the combination of weighted-sum layer fusion, a brief end-to-end fine-tuning of the SSL weights (5,000 steps, then frozen to avoid catastrophic forgetting), and training-time mixing of background music into the vocal data. Weight visualizations show that before fine-tuning only shallow layers encode melody, whereas fine-tuning shifts substantial melody information into higher transformer layers, which the authors connect to the large F0RMSE gains from 0.360 (single-layer HuBERT) to 0.176 (proposed WavLM).","core_discovery":"On the paper's own terms, the discovery is that a 256-dimensional melody representation extracted from a fine-tuned self-supervised model can replace an explicit pitch contour as the melodic conditioning in any-to-one singing voice conversion, and that this change is what makes conversion handle accompaniment. The melody extractor computes a learnable weighted sum of all hidden-layer outputs of WavLM (or HuBERT), $O=\\sum_{l=0}^{L} w_l h_l^t$, feeds this through Feed Forward Transformer blocks that predict pitch, energy, and voiced/unvoiced flags, and uses the resulting hidden state as the melody feature. The authors train the extractor on vocal tracks mixed with background music, fine-tune the SSL weights for only 5,000 steps, then freeze them. In BGM conditions the proposed system achieves the best F0RMSE/F0CORR at every SNR tested; under clean audio its objective metrics are comparable to the baselines, while its listener-rated naturalness and similarity are higher. At 0 dB the WavLM system reaches F0RMSE 0.199 and F0CORR 0.935. The paper also claims that fine-tuning is what activates melody information in the upper transformer layers: weight visualisations show that before fine-tuning only the CNN and lower encoder layers contribute, while after fine-tuning the higher layers contribute substantially.","pith_inferences":["A natural extension the paper does not test is whether the same melody feature transfers to any-to-many or multi-singer conversion, since the proposed model is any-to-one; the melody extractor itself is singer-independent and should be portable.","Because the objective pitch metric is computed against F0 trackers of the same family used to create training labels, a stronger test would use a synthetic mixture with a known musical ground truth (for example, MIDI-rendered accompaniment and a click-track performance) to see whether the apparent gain reflects true pitch or merely alignment to tracker behavior.","The 0 dB condition is the paper's most dramatic improvement; pushing SNR lower or adding nonstationary noise such as applause, speech, or competing vocals would reveal whether the SSL representations eventually hit the same failure floor as conventional trackers.","The weight visualizations suggest a testable prediction: a fixed pretrained SSL model with no fine-tuning should have most melody information in shallow layers, so a lightweight probe on those layers may approximate the proposed extractor at lower computational cost."],"forward_implications":["An SVC pipeline with this melody extractor can accept a full song with accompaniment as input and output a clean vocal in the target singer's timbre, so users do not need to run source separation first.","At 0 dB SNR the claimed melody accuracy (F0RMSE 0.199, F0CORR 0.935) is better than both the neural tracker Crepe and the separated+Crepe pipeline, meaning the gain comes from the representation, not from cleaning the audio.","Because the feature carries energy and voiced/unvoiced information, converted audio retains more of the source's expressive timing and articulation than a scalar F0 contour would provide.","The extractor generalizes across two different SSL architectures (HuBERT and WavLM), and WavLM's stronger results suggest that the choice of pretrained model matters more than the downstream head design."],"supporting_citations":[{"why":"Supplies the HuBERT SSL model whose hidden layers are compared in melody extraction.","marker":"[20]"},{"why":"Supplies the WavLM SSL model used in the proposed melody extractor.","marker":"[21]"},{"why":"Provides the recognition-synthesis SVC backbone that the proposed method extends.","marker":"[22]"},{"why":"Crepe is the neural F0 extractor used as a baseline and beaten at low SNR.","marker":"[16]"},{"why":"Demucs is the source-separation baseline used in Separated+Crepe, which the method avoids.","marker":"[35]"},{"why":"Defines the F0RMSE/F0CORR objective metrics used to evaluate melody preservation.","marker":"[36]"},{"why":"Provides the weighted-sum technique for combining SSL hidden-layer outputs.","marker":"[27]"},{"why":"Supplies the accompanied-music training data and test mixtures with BGM.","marker":"[28]"},{"why":"Supplies the clean vocal corpus used to train the melody extractor and as out-of-domain SVC data.","marker":"[29]"},{"why":"Supplies the Opencpop in-set target-singer corpus for SVC training.","marker":"[30]"}],"fun_headline_variants":["SSL melody features keep singing conversion on pitch under loud BGM","No source separation: self-supervised melody features make SVC robust to BGM","Fine-tuned SSL melody features cut pitch error 33% at 0 dB BGM","Self-supervised melody features improve singing conversion under background music"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the ground-truth melody being the median pitch reported by three software trackers (PYIN, REAPER, Parselmouth) on the clean vocal; if those trackers are systematically wrong for expressive singing or dense accompaniment, the reported pitch-accuracy improvements could partly reflect better matching of the trackers rather than better melody.","fun_headline_variants_meta":{"raw":{"variants":["SSL melody features keep singing conversion on pitch under loud BGM","No source separation: self-supervised melody features make SVC robust to BGM","Fine-tuned SSL melody features cut pitch error 33% at 0 dB BGM","Self-supervised melody features improve singing conversion under background music"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00074,"raw_usage":{"total_tokens":3349,"prompt_tokens":1033,"completion_tokens":2316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":2236}},"tokens_in":649,"tokens_out":2316,"duration_ms":17887,"temperature":1.0,"reasoning_tokens":2236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:42:30.383129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of songs with a known reference melody (for example, synthesized or performed against a precise MIDI score), mix in accompaniment at 0 dB SNR, convert with the proposed WavLM system, and compare the converted pitch to the known notes rather than to a tracker-derived contour; if the proposed method does not keep its advantage over Crepe and Separated+Crepe against this reference, the central claim of superior melody accuracy fails.","supporting_citations":[{"cited_title":"RobustSVC: HuBERT-based Melody Extractor and Adversarial Learning for Robust Singing Voice Conversion","cited_arxiv_id":"2409.06237","evidence_quote":"Provides the recognition-synthesis SVC backbone that the proposed method extends."},{"cited_title":"Crepe: A convolutional representation for pitch estimation,","cited_arxiv_id":null,"evidence_quote":"Crepe is the neural F0 extractor used as a baseline and beaten at low SNR."},{"cited_title":"Hybrid transformers for music source separation,","cited_arxiv_id":null,"evidence_quote":"Demucs is the source-separation baseline used in Separated+Crepe, which the method avoids."},{"cited_title":"The singing voice conversion challenge 2023,","cited_arxiv_id":null,"evidence_quote":"Defines the F0RMSE/F0CORR objective metrics used to evaluate melody preservation."},{"cited_title":"Why does self-supervised learning for speech recognition benefit speaker recognition?","cited_arxiv_id":null,"evidence_quote":"Provides the weighted-sum technique for combining SSL hidden-layer outputs."},{"cited_title":"Musdb18-a corpus for music separation,","cited_arxiv_id":null,"evidence_quote":"Supplies the accompanied-music training data and test mixtures with BGM."},{"cited_title":"Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus,","cited_arxiv_id":null,"evidence_quote":"Supplies the clean vocal corpus used to train the melody extractor and as out-of-domain SVC data."}],"review_version":1}