{"id":"6b726c02-2d59-4314-88d2-86d53a8c1751","arxiv_id":"2505.09325","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SingNet is a claimed ~3,000-hour in-the-wild singing voice dataset from internet songs and sample packs, with benchmarks for lyric transcription, vocoders, and singing voice conversion.","lead":"The authors introduce SingNet, a pipeline and claimed 3,000-hour multilingual, in-the-wild singing voice dataset gathered from internet songs and sample packs, with pretrained vocoder, SSL, and SVC models. It matters because data scale and diversity are the main bottlenecks for singing voice synthesis and conversion, and this is an order of magnitude larger than prior public singing datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test leakage in the in-the-wild SVC evaluation is plausible and unaddressed; the SingNet-SS test sample appears to be drawn from the same pipeline that generated the training mixture.","rationale":"The reader identified the same weakest assumption: the in-the-wild evaluation set is sampled from SingNet-SS, the same source as part of the training mixture, and no disjointness statement is given. I agree with the CONDITIONAL verdict because the dataset and pipeline are not yet released, making the central claim unverifiable, and the benchmark design has a plausible leakage path. However, the paper does have independent support: the ALT experiment tests on SingStyle111 (disjoint from SingNet-SS) and shows a WER improvement from 7.79% to 6.76%; the vocoder copy-synthesis results show consistent MCD gains on the In-the-wild test set. The most load-bearing vulnerability is specifically the SVC table (Table 4): the three SVC runs differ only in training scale, the In-the-wild test set is drawn from the same source as the largest training set, and the subjective MOS/SMOS improvements have overlapping confidence intervals on several rows. The proposed nearest-neighbor overlap check is feasible without full release and would settle whether the scaling benefit is confounded by train/test overlap. If the check comes back clean, the CONDITIONAL verdict could be upgraded; if not, the SVC scaling claim needs to be walked back.","tokens_in":14130,"tokens_out":1857,"duration_ms":17642,"concrete_test":"Check for overlap between the SingNet-SS test split (the 6-hour sample) and the training split used for SVC/vocoder training. Concretely: compute embeddings (e.g., WavLM or MERT) for all training and test utterances, run nearest-neighbor search with a threshold calibrated on known duplicates, and also compare song-level metadata if available. If any test utterance is within a duplicate-detection threshold of a training utterance, rerun the SVC and vocoder evaluations on a split that excludes all segments from the same song/recording. Report the WER/CER/MOS/SMOS differences; if the gaps in Table 4 remain, the benchmark supports the scaling claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that SingNet is useful as large-scale in-the-wild training data rests on the SVC, vocoder, and ALT benchmarks. The weakest load-bearing point is in Section 4.1.1: the In-the-wild evaluation set is a 6-hour random sample from SingNet-SS, which is also a component of the 3500-hour mixture used to train the SVC and vocoder systems. The paper never states that the evaluation sample is disjoint from the training data, nor does it describe a deduplication or exclusion procedure. SingNet-SS is constructed from internet songs, and the pipeline includes lyric-based segmentation and audio restoration; even if the sampled test utterances are unique segments, the same songs, singers, and recording environments may appear in both sets. Since SVC models can generalize to unseen singers but are prone to overfitting to seen recordings, and the paper's SVC scaling result is largely driven by subjective MOS/SMOS gaps that overlap in CI, a hidden overlap could inflate the apparent benefit of SingNet-SS. This is not an accusation of deliberate contamination; it is an unverified experimental control. The same concern applies to the vocoder In-the-wild test points, though the vocoder result is more robust because the baseline Large-Compilation also sees the test utterances.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SingNet proposes a pipeline for extracting large-scale in-the-wild singing voice data from internet songs and sample packs, yielding around 3,000 hours of multilingual, multi-style data. The authors pre-train Wav2vec2, BigVGAN, and NSF-HiFiGAN on a 3,500-hour singing mixture and report benchmarks for automatic lyric transcription, neural vocoding, and zero-shot singing voice conversion. They argue that their dataset and open-sourced checkpoints outperform existing smaller or speech/general-audio-based training sets, and that scaling up singing data improves downstream performance.","tokens_in":14376,"tokens_out":4297,"duration_ms":43526,"significance":"If the dataset, pipeline, and checkpoints are released and the benchmark claims are validated with clean train/test separation, SingNet would be a valuable community resource: it is the largest in-the-wild singing dataset to date, the pipeline is an original contribution, and the pre-trained models provide useful baselines. The paper also makes a falsifiable scaling claim about singing voice data. However, the current experimental design does not yet support these claims, because the evaluation sets appear to overlap with the training mixture and the subjective scaling results are statistically inconclusive.","major_comments":[{"comment":"The evaluation sets are not shown to be disjoint from the training data. The 3,500-hour training mixture is described as including all existing singing datasets, which includes SingStyle111, and also includes SingNet-SS (Section 4.1.1). The Studio Recording evaluation is a random 1-hour sample from SingStyle111, and the In-the-wild evaluation is a 6-hour random sample from SingNet-SS. The paper never states that these samples were excluded from training, nor does it describe deduplication at the song, singer, or recording level. If even part of the test material appears in the training mixture, Tables 2-4 may reflect memorization rather than generalization. This is a load-bearing issue for the central claim that SingNet helps downstream tasks, and it also extends to the ALT and vocoder experiments. The authors must specify the exclusion/deduplication procedure and ideally rerun the experiments on strictly held-out material.","section":"Section 4.1.1 and Section 4.1.3"},{"comment":"The SVC scaling claim is not supported by the reported confidence intervals. In the Studio Recording setting, MOS is 3.72±0.17 for 35h and 3.73±0.19 for 3500h, and SMOS is 3.55±0.25 vs 3.71±0.27; in the In-the-wild setting, MOS is 3.16±0.16 for 3500h versus 3.01±0.17 for 350h, and SMOS is 3.11±0.25 vs 2.98±0.24. All pairs overlap within their 95% intervals. Moreover, the 350h condition is non-monotonic (worse than the 35h condition on several metrics), yet the text asserts that the 3500h system 'performs the best'. Stating that 'the MOS scores are within 95% CI' is not the same as showing a significant difference. The claim that scale alone drives SVC quality needs significance testing, more listeners/utterances, or a different evaluation design.","section":"Table 4 and Section 4.5"},{"comment":"The paper is a dataset paper but never states whether the SingNet audio, annotations, and pipeline code will be released, nor under what license. The only explicit release commitment is for checkpoints 'after the double-blind review period' (Section 1). Without a clear data-release plan, the dataset contribution cannot be used or independently verified, and the paper's central artifact remains inaccessible. Please add a data availability statement covering the audio, the sample-pack captions, the pipeline scripts, and the license/usage terms, including any copyright or redistribution restrictions on the crawled internet audio.","section":"Data availability and licensing"}],"minor_comments":[{"comment":"In the contribution bullet list, 'in-the-wide' should be 'in-the-wild'.","section":"Section 1 (contributions)"},{"comment":"The diversity claim is supported only by a qualitative PCA scatter plot. A quantitative measure of coverage or dispersion (e.g., estimated volume, entropy, or nearest-neighbor statistics) would strengthen the claim that SingNet is more acoustically and semantically diverse than the existing-dataset mixture.","section":"Section 3.3 / Figure 4"},{"comment":"The MOS-based filtering threshold of 3.0 appears to be a hand-chosen value. A sentence justifying this choice, or a brief sensitivity analysis showing how downstream performance varies with the threshold, would make the pipeline more reproducible.","section":"Section 3.1.3"},{"comment":"In the Studio Recording setting for BigVGAN, adding SingingVoice-3500 improves MCD (1.777 to 1.520) but worsens F0RMSE (35.897 to 37.125). The text claims the singing-voice-trained system 'significantly outperformed' the Large-Compilation system; this should be qualified because the F0 metric moves in the opposite direction.","section":"Table 3"},{"comment":"The 'manual review' of synthesized samples is anecdotal and should be described as such, not used as primary evidence for the scaling claim. If quantitative support exists (e.g., error analysis on a subset), it should be reported systematically.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The dataset-construction effort is substantial and the paper has a clear contribution, but the experimental validation currently has a train/test overlap problem that could invalidate the headline results, and the SVC scaling claim is statistically weak. The overlap issue is not merely stylistic; it must be fixed by re-running benchmarks on explicitly held-out data. I also flag the missing statement on data release and licensing, which is essential for a dataset paper. This is a major revision rather than a reject because the central idea is sound and the paper can be repaired within its scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SingNet is the largest in-the-wild singing dataset by an order of magnitude, and the pipeline is genuinely useful. But the SVC benchmark looks like it may have train/test leakage, and the resources aren't out yet, so treat the empirical claims with caution.\n\nWhat's new: the scale is real - 3000 hours from internet songs and sample packs, versus 322 hours for ACESinger. The pipeline is described in enough detail to reproduce: MDX23 source separation, a chain of VST restoration plugins, lyric-based segmentation, VAD, multi-F0 rejection, and a SingMOS-trained quality filter at threshold 3.0. The semi-supervised caption system for sample packs is a clever way to turn messy producer folders into labeled dry/wet vocal categories. That's a contribution independent of the dataset itself.\n\nThe ALT experiment is the cleanest result: pre-training Wav2vec2 on singing voice beats speech transfer learning for low-resource lyric transcription. That's a useful datapoint.\n\nWhere it gets shaky: the stress-test note is right. Section 4.1.1 says the in-the-wild evaluation set is a random sample from SingNet-SS, which is a component of the 3500-hour mixture used to train the SVC and vocoder models. The paper never says the test sample is excluded from training, and there's no deduplication described. Since the 3500h condition includes the test songs and possibly the exact segments, the SVC and vocoder numbers may be inflated. This isn't an accusation; it's an unverified experimental control that the authors need to fix. The SVC scaling story is also weak on its own: the MOS gap between 35h and 3500h in the studio setting is within confidence intervals, and the 350h condition is non-monotonic. The diversity analysis is a PCA plot with no quantitative measure. And the abstract says \"open-source\" but the footnote says checkpoints come after the review period; the dataset itself isn't linked. So the central claim of usefulness is currently unverifiable.\n\nThe citation pattern is fine; Emilia is the natural template and they extend it to singing, which is harder.\n\nMy recommendation: this deserves a serious referee, not a desk reject. The dataset could be a community resource, and the pipeline should be published even if the experimental section needs work. But I'd ask for the data, code, and checkpoints to be released, an explicit train/test disjointness statement or a deduplication step, and a more careful SVC analysis that doesn't claim scaling from overlapping CIs. If the leakage is real, the scaling claim collapses; if the data is released, this becomes a reference dataset.","headline":"SingNet is the largest in-the-wild singing dataset with a genuinely useful pipeline, but a likely train/test overlap and unreleased resources weaken the empirical claims.","tokens_in":14909,"tokens_out":3104,"would_cite":false,"duration_ms":29407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an open pipeline can turn internet songs and sample packs into about 3,000 hours of usable singing voice data, and that pre-training on this corpus improves lyric transcription, vocoders, and singing voice conversion.","keywords":["SingNet","singing voice dataset","in-the-wild audio","data processing pipeline","source separation","automatic lyric transcription","neural vocoder","singing voice conversion"],"falsifier":"Run a near-duplicate audio search between the 6-hour SingNet-SS evaluation set and the 3,500-hour training mixture; if any exact or near-exact segments appear, the reported ALT/vocoder/SVC gains could come from memorization rather than from dataset scale. A second concrete check is a blind listening test in which experienced listeners mark artifacts on clips that scored just above 3.0.","tokens_in":13944,"feed_emoji":"🎤","tokens_out":6553,"duration_ms":60358,"temperature":0.7,"pith_summary":"SingNet tries to break the data bottleneck that has kept singing voice synthesis and conversion tied to small, expensive studio recordings. The authors build an automated, open-source pipeline that mines about 3,000 hours of singing voice from internet songs and sample packs, spanning many languages and styles. They then pre-train Wav2vec2, BigVGAN, and NSF-HiFiGAN models on this data and benchmark them on automatic lyric transcription, neural vocoding, and singing voice conversion. If the pipeline and benchmarks hold up, singing voice research gains a scalable, renewable data source and open checkpoints that do not require professional studio recording.","feed_headline":"Internet songs become 3,000 hours of singing-voice training data","feed_subtitle":"Open checkpoints show gains in lyric transcription, vocoder naturalness, and voice-conversion similarity.","key_machinery":"The load-bearing mechanism is the open data-processing pipeline. It separates vocals from songs with the MDX23 model, runs a chain of restoration plugins to remove noise, reverb, clicks, and plosives, segments audio by lyric timestamps or voice activity detection, rejects multi-singer clips via multi-F0 detection, and keeps only clips scored above 3.0 by a fine-tuned MOS predictor. For sample packs, a semi-supervised captioning system trains an audio classifier that separates acapella and adlibs from elements and FX. Pre-trained checkpoints built on this data serve as the benchmark instruments for ALT, vocoder, and SVC experiments.","core_discovery":"The central discovery claimed is that the singing-voice data scarcity bottleneck can be addressed without studio recording: a pipeline of source separation, audio restoration, segmentation, multi-F0 rejection, and MOS-based quality filtering yields about 3,000 hours of multilingual, multi-style singing voice data (2,629 hours from in-the-wild songs and 321 hours from sample packs). Pre-training on this mixture, optionally combined with existing singing datasets to form 3,500 hours, produces models that outperform or match existing open-sourced models on automatic lyric transcription, neural vocoder quality, and singing voice conversion benchmarks.","pith_inferences":["If SingNet's scale and cleanliness hold up, singing-voice synthesis could follow the speech path where web-scale data plus robust pre-training replaces studio-perfect corpora; the released checkpoints make this testable by any group.","The curious SVC pattern where 35 and 3,500 hours beat 350 hours suggests an intermediate-data regime in which encoder capacity lags diversity; a testable extension is to vary encoder size with data diversity to see whether the dip disappears.","A direct test of the MOS filter's role would be to train the vocoder and SVC on unfiltered versus filtered SingNet subsets; if gains vanish without filtering, the quality gate is the real driver rather than raw hours.","The sample-pack classifier could become a standalone tool for mining studio-quality acapella stems at scale, labeling newly released packs as dry or wet acapella and adlibs without human captions."],"forward_implications":["Singing-voice self-supervised pre-training makes ALT fine-tuning possible without speech-based transfer learning: the SingNet-trained Wav2vec2 reaches 6.76% WER versus 7.79% for the transfer-learning approach.","Vocoders trained on 3,500 hours of singing voice clearly beat speech/general-sound-only baselines on in-the-wild audio (BigVGAN MOS 3.55 vs 3.13; NSF-HiFiGAN MOS 3.52 vs 3.39).","SVC subjective quality and similarity improve at the 3,500-hour scale (studio MOS 3.73 and SMOS 3.71; in-the-wild MOS 3.16 and SMOS 3.11), although objective F0 and intelligibility metrics are roughly flat across scales.","The dataset can be extended over time by running the released pipeline on more sources, making scale a renewable resource rather than a fixed studio corpus."],"supporting_citations":[{"why":"Supplies the in-the-wild speech data-scaling recipe and open-pipeline approach on which SingNet is modeled.","marker":"He et al. (2024)"},{"why":"Provides the sound demixing tool/library used for vocal source separation.","marker":"Solovyev et al. (2023)"},{"why":"Provides the MDX23 pre-trained model used to extract wet vocal stems from songs.","marker":"Fabbro et al. (2024)"},{"why":"Method used to fine-tune a singing-voice MOS predictor for quality filtering.","marker":"Cooper et al. (2022)"},{"why":"Supplies the SingMOS dataset on which the quality scorer is fine-tuned.","marker":"Tang et al. (2024)"},{"why":"Multi-F0 detection used to reject clips with multiple simultaneous singers.","marker":"Cuesta et al. (2020)"},{"why":"BigVGAN baseline and the Large-Compilation training distribution used in vocoder benchmarks.","marker":"Lee et al. (2023)"},{"why":"NSF-HiFiGAN baseline and the DiffSinger implementation used for vocoder training and benchmarking.","marker":"Liu et al. (2022a)"},{"why":"Previous ALT transfer-learning approach whose WER SingNet's direct-tune result is compared against.","marker":"Ou et al. (2022)"},{"why":"Prior data-scaling effort whose manual tuning cost motivates SingNet's automated pipeline.","marker":"Shi et al. (2024)"}],"fun_headline_variants":["Internet songs yield 3,000-hour singing dataset","SingNet: 3,000 hours of in-the-wild singing data","Web songs become 3,000-hour singing voice dataset","3,000 hours of singing voices from internet songs","SingNet: Harvesting 3,000 hours of singing from the web"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark results assume the MOS quality gate (>3.0) and the separation/restoration chain leave no audible artifacts, and that the 6-hour in-the-wild evaluation set contains no actual training audio from the 3,500-hour mixture.","fun_headline_variants_meta":{"raw":{"variants":["Internet songs yield 3,000-hour singing dataset","SingNet: 3,000 hours of in-the-wild singing data","Web songs become 3,000-hour singing voice dataset","3,000 hours of singing voices from internet songs","SingNet: Harvesting 3,000 hours of singing from the web"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3644,"prompt_tokens":840,"completion_tokens":2804,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":2717}},"tokens_in":456,"tokens_out":2804,"duration_ms":19336,"temperature":1.0,"reasoning_tokens":2717,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:34:12.566871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a near-duplicate audio search between the 6-hour SingNet-SS evaluation set and the 3,500-hour training mixture; if any exact or near-exact segments appear, the reported ALT/vocoder/SVC gains could come from memorization rather than from dataset scale. A second concrete check is a blind listening test in which experienced listeners mark artifacts on clips that scored just above 3.0.","supporting_citations":[{"cited_title":"Benchmarks and leaderboards for sound demixing tasks, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the sound demixing tool/library used for vocal source separation."},{"cited_title":"Generalization Ability of MOS Prediction Networks","cited_arxiv_id":null,"evidence_quote":"Method used to fine-tune a singing-voice MOS predictor for quality filtering."},{"cited_title":"Multiple F0 Estimation in Vocal Ensembles using Convolutional Neura l networks","cited_arxiv_id":null,"evidence_quote":"Multi-F0 detection used to reject clips with multiple simultaneous singers."},{"cited_title":"BigVGAN: A Universal Neural Vocoder with Large-Scale Training","cited_arxiv_id":null,"evidence_quote":"BigVGAN baseline and the Large-Compilation training distribution used in vocoder benchmarks."}],"review_version":1}