{"id":"f4d70b1c-a994-4dc6-925f-4e073827c1f2","arxiv_id":"2502.02366","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SSL convolutional audio models pre-trained on speech, non-speech, or both perform nearly equally well across speech and non-speech downstream tasks, while domain-specific baselines struggle outside their domains.","lead":"This paper tests whether self-supervised audio models pre-trained on speech, non-speech, or both generalize across audio domains. It finds that the pre-training diet matters little, and that such models transfer better than popular domain-specific baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'non-speech' pre-training diet is contaminated with speech (MUSAN and AudioSet), so the claim that pre-training domain only weakly affects downstream performance is not cleanly tested; observed similarities may partly reflect speech leakage.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern I find: the 'non-speech' pre-training corpus is not cleanly non-speech because MUSAN and AudioSet contain speech. This is not a matter of outside-consensus interpretation; it is a factual mismatch with the paper's own cited references. The central claim that pre-training domain has little effect on downstream performance depends on the two single-domain diets being well separated. With speech leaking into the non-speech diet, the BYOL-A(Non-Speech) model becomes a de facto mixed-diet model, so its strong speech-task performance and its similarity to BYOL-A(Speech) are expected even if domain specificity were large. The paper does not report per-class breakdowns or contamination checks, and the limitations section does not mention this. That said, the overall flexibility claim has independent support: the speech-only model performs well on clearly non-speech held-out tasks, and the model trained on all data also transfers broadly. The issue is addressable by data auditing and retraining, and the public checkpoints make this feasible. Therefore I do not recommend changing the reader's conditional verdict; the concern strengthens the conditions rather than forcing rejection. I agree with the reader's assessment that confirming the main result under a clean non-speech diet would make the paper a solid empirical contribution.","tokens_in":10920,"tokens_out":3000,"duration_ms":33185,"concrete_test":"Audit the exact training files used for the non-speech model: in the public checkpoint repository, identify whether MUSAN speech files and AudioSet clips with speech-related labels (e.g., 'Speech', 'Human voice') were included. Compute the fraction of non-speech corpus hours that are speech. If the fraction is non-negligible, retrain a clean non-speech BYOL-A model after excluding all speech-containing clips and rerun the Figure 1 transfer experiments for Speech Commands, VCTK, and AVA-Speech. If the clean non-speech model's accuracy drops by more than the small gaps reported among BYOL-A diets, or if the BYOL-A(Speech) vs BYOL-A(Non-Speech) gap widens substantially, the domain-specificity conclusion is confounded; if performance remains within the same small range, the conclusion survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that BYOL-A models show only small domain-specificity advantages rests on the assumption that the 'non-speech' pre-training corpus is actually non-speech. This assumption fails. Table I in Section II.A lists MUSAN and AudioSet as non-speech. MUSAN, by its own description (Music, Speech, and Noise corpus, ref [26]), contains a dedicated speech subset. AudioSet (ref [25]) includes numerous speech-related classes such as 'Speech' and 'Human voice'; the paper does not state that those clips were removed. AudioSet dominates the non-speech diet (4,817 of roughly 5,200 non-speech hours), so even a small speech fraction represents hundreds of hours. Consequently, the non-speech BYOL-A model likely saw substantial speech during pretraining, making its strong performance on Speech Commands, VCTK, and AVA-Speech less surprising and inflating the apparent similarity among diets. The paper's own limitations section (Section IV) does not acknowledge this contamination. This confound directly affects the domain-specificity comparison, though it does not entirely invalidate the flexibility claim: the speech-only model's performance on held-out non-speech tasks (ESC-50, UrbanSound8K, NSynth) remains evidence of cross-domain flexibility. The key unsupported piece is the claim that 'only small domain-specificity advantages were observed' between the speech and non-speech pre-training datasets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether the domain of self-supervised pre-training data affects the downstream transfer performance of a convolutional BYOL-A audio model. The author trains three BYOL-A models from scratch on roughly 5,000-hour corpora of speech-only, non-speech-only, and combined audio, then evaluates frozen embeddings via linear probing on seven held-out tasks spanning speech, non-speech, and voice-activity detection, comparing against YAMNet and WavLM baselines. Representational similarity analysis is used to compare the learned embedding spaces across diets. The central claim is that all three BYOL-A models perform well across nearly all tasks with only small domain-specificity advantages, suggesting that SSL convolutional audio models learn flexible representations despite domain mismatch.","tokens_in":11143,"tokens_out":4134,"duration_ms":40253,"significance":"If the conclusion were fully supported, the paper would provide a practical and useful message for transfer learning in audio: that SSL pre-training on either speech or non-speech data may yield representations flexible enough for downstream tasks in both domains. The study uses a clean linear-probing protocol, standard public benchmark datasets, and publicly releases the three pre-trained checkpoints, which are concrete contributions that support reproducibility. The RSA comparison to acoustic descriptors is also a thoughtful addition. However, the central domain-specificity comparison is weakened by a testable confound in the construction of the non-speech pre-training corpus, and the 'small differences' claim lacks uncertainty quantification.","major_comments":[{"comment":"The non-speech pre-training corpus is contaminated with speech, which confounds the central comparison. MUSAN (ref. 26) is by its own name a Music, Speech, and Noise corpus and contains a dedicated speech subset, while AudioSet (ref. 25) includes speech-related labels such as 'Speech' and 'Human voice'; the paper does not state that these clips were removed. Since AudioSet contributes 4,817 of the roughly 5,200 non-speech hours, even a small speech fraction corresponds to hundreds of hours. Consequently, the BYOL-A model labeled 'Non-Speech' likely saw substantial speech during pre-training, and its strong performance on Speech Commands, VCTK, and AVA-Speech is not clean evidence that pre-training domain has little effect. The claim in Section III that 'the pre-training domain only had a small influence on downstream task performance' is therefore not cleanly tested; the author should either re-train the non-speech model on a verified speech-free corpus or clearly qualify the result.","section":"II.A, Table I"},{"comment":"The paper reports no uncertainty or significance testing for the differences among the three BYOL-A diets. The conclusion that only 'small' domain-specificity advantages exist appears to be based on visual inspection of single runs. Because pre-training involves stochasticity, and because the claim is specifically about the size of a difference, the paper should provide multiple seeds with confidence intervals or a statistical comparison of the accuracy distributions, at least for the key tasks where the differences are visually small.","section":"III, Fig. 1"},{"comment":"The limitations section does not acknowledge the speech contamination in the non-speech pre-training corpus. Given that this is a direct threat to the main claim about domain specificity, it should be stated explicitly and discussed, along with the potential impact on the interpretation of the RSA results.","section":"IV"}],"minor_comments":[{"comment":"The paper uses 'Yamnet' and 'YAMNet' inconsistently; standardize the model name. Also, in Figure 1 the label 'Y amnet' contains an erroneous space.","section":"Global"},{"comment":"The paper references 'Supplemental Table 1' and 'Supplemental Figures 1 and 2' but these are not included in the manuscript; ensure they are available with the submission.","section":"References to supplements"},{"comment":"The decision to retain only correlations with rs > 0 before FDR correction is unusual and should be justified; a positive threshold can bias the average correlations upward.","section":"II.D"},{"comment":"Minor grammatical issues: e.g., 'a sounds' overall amplitude' should be 'a sound's overall amplitude'.","section":"III"},{"comment":"The acronym 'BYOL-A1' in the text should be 'BYOL-A'; the footnote marker is not needed in the running text.","section":"II.B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an audio/machine-learning journal and makes a reproducible contribution via released checkpoints. The main issue is the contamination of the non-speech pre-training corpus, which directly affects the paper's headline claim about small domain-specificity effects. This is fixable by re-training on a verified speech-free corpus or by substantially qualifying the claim. The lack of multiple runs is also a concern but may be acceptable for a single-author empirical study; however, the journal should expect at least confidence intervals or a clear statement about single-run variability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful empirical study, but the central claim is softer than the abstract suggests. The author trained BYOL-A convolutional models on speech-only, non-speech-only, and combined audio diets (roughly matched in size), then linear-probed frozen embeddings on six held-out tasks spanning speech, non-speech, and VAD. The protocol is clean, the checkpoints are public, and the RSA is a nice addition. The most convincing result is that the speech-only model transferred well to non-speech tasks (ESC-50, UrbanSound8K, NSynth) and the non-speech model to speech tasks; that is real evidence of flexibility, and it matters for practitioners who want to skip matched-data collection.\n\nThe soft spot is exactly where the stress test lands. The 'non-speech' diet is not actually non-speech. MUSAN contains a speech subset by definition, and AudioSet, which dominates the diet (4,817 of ~5,200 hours), includes speech and voice classes. The paper does not report filtering those out. So the non-speech model almost certainly saw hundreds of hours of speech during pre-training. That makes the finding that all three models perform similarly on speech tasks less surprising, and it directly undermines the 'only small domain-specificity advantages' claim. The flexibility claim survives, because the speech-only model's non-speech performance cannot be explained by contamination, but the symmetry of the result is overstated.\n\nTwo smaller issues. There are no error bars or multiple seeds, so 'small differences' are hard to evaluate; a single run on a fixed train/test split does not support a claim about the size of an advantage. And the limitations section mentions transformer architectures, classification-only tasks, and scale, but never the data contamination. That should be acknowledged.\n\nBottom line: this deserves a serious referee, and I would engage with it. The fixes are concrete: re-categorize or filter the non-speech data, or retrain the diets; add repeated runs with confidence intervals; and revise the abstract's claim accordingly. The paper is a solid empirical contribution once the confound is handled, and the representational similarity analysis is a useful piece of evidence about convergence across diets.","headline":"Useful and reproducible study of BYOL-A pre-training diets, but the 'non-speech' diet is contaminated with speech, so the domain-specificity claim needs a cleaner test.","tokens_in":11718,"tokens_out":2465,"would_cite":false,"duration_ms":24794,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The domain of pre-training data has only a small effect on how well a self-supervised audio model transfers to downstream speech and non-speech tasks.","keywords":["self-supervised learning","audio representation learning","transfer learning","domain specificity","BYOL-A","representational similarity analysis","sound event recognition","speech recognition"],"falsifier":"Train a BYOL-A model on a rigorously speech-free subset of the non-speech corpora (e.g., AudioSet clips with no speech-related labels, MUSAN noise and music only) and a matched speech-only model, then compare their transfer accuracy on a speech task like Speech Commands. If the clean non-speech model loses most of its speech-task performance, the paper's flexibility conclusion would be substantially weakened.","tokens_in":10643,"feed_emoji":"🎧","tokens_out":5111,"duration_ms":43838,"temperature":0.7,"pith_summary":"This paper asks whether the domain of a self-supervised audio model's pre-training data limits what tasks it can later be transferred to. Using the BYOL-A convolutional architecture, the author pre-trains three models on matched volumes of speech, non-speech, and combined audio, then evaluates each on held-out speech, non-speech, and voice-activity-detection tasks. The central finding is that pre-training domain has only a small effect: all three models perform well on nearly all tasks, often matching or beating domain-specific baselines such as Yamnet and WavLM in their own or opposite domains. The paper interprets this as evidence that SSL convolutional models learn flexible acoustic features whose transfer value does not strongly depend on matching the pre-training diet to the target domain.","feed_headline":"Pre-training domain barely matters for self-supervised audio models","feed_subtitle":"BYOL-A models trained on speech, non-speech, or both match or beat domain-specific baselines on most tasks.","key_machinery":"The load-bearing mechanism is BYOL-A ('bootstrap your own latent for audio'), a self-supervised method that applies random audio augmentations—noise, cropping, resizing, time-frequency stretching, amplitude modulation—to spectrograms and trains a two-block convolutional network to output representations invariant to those manipulations. The transfer protocol then freezes the pre-trained embeddings and learns only a linear layer on top for each downstream task, so downstream performance reflects the quality of the pre-trained features themselves. Representational similarity analysis (RSA) is the interpretative tool: it correlates pairwise distances among a model's embeddings with pairwise differences along acoustic descriptor dimensions, revealing which features the models encode.","core_discovery":"The paper claims that BYOL-A models pre-trained on speech data, non-speech data, or both converge on similar, broadly useful acoustic representations. Across held-out transfer tasks—keyword recognition, speaker recognition, sound event recognition, instrument recognition, and voice activity detection—the three models' accuracies cluster together, with only small advantages for diet-matched tasks. The domain-specific baselines behave differently: Yamnet excels on non-speech tasks but fades on speech, and WavLM excels on speech but struggles on non-speech. Representational similarity analysis shows the BYOL-A models' embedding spaces are highly correlated with one another (Spearman $\\rho = 0.88$–$0.95$) and share sensitivity to spectral energy in frequency bands above 600 Hz, spectral variability, overall amplitude, and pitch. The author concludes that this SSL approach yields flexible feature learners for which domain mismatch between pre-training and downstream data is not a major penalty, at least for classification-style transfer.","pith_inferences":["The convergence of representational spaces across diets hints that BYOL-A's augmentation set imposes a stronger inductive bias than the data distribution; one testable extension would be to ablate individual augmentations and measure how much cross-domain transfer survives.","Scale may interact with this result: the paper itself notes WavLM used far more data; at larger scales or with transformer architectures, domain specificity could re-emerge, especially on sequence-level or regression tasks, which were not tested here.","The results suggest a practical recipe for data-exploration tools: a single BYOL-A embedding space can serve outlier detection, sound synthesis steering, or bioacoustic monitoring without retraining, provided the target features align with the spectral/source character the model encodes.","A stronger test of flexibility would be few-shot or zero-shot probing—evaluating the frozen embeddings with no fine-tuning—where differences between diets might be more visible than after a learned linear layer."],"forward_implications":["Practitioners can pre-train one SSL audio model on whatever unlabeled audio is abundant and expect it to transfer across speech and non-speech classification tasks, reducing the need for domain-matched pre-training data.","Domain-specific supervised baselines remain valuable in their home domain, but their out-of-domain brittleness makes SSL models a safer default when the downstream domain is uncertain or mixed.","The high inter-model representational similarity ($\\rho = 0.88$–$0.95$) suggests that pre-training diet is not the main determinant of what a BYOL-A model learns; augmentation strategy and architecture may matter more.","For voice activity detection, all models, including those with little or no speech pre-training, performed well even in the data-limited regime, indicating VAD features are learnable from generic audio."],"supporting_citations":[{"why":"Supplies the BYOL-A self-supervised pre-training method that the paper trains from scratch on its three audio diets.","marker":"[11,12]"},{"why":"Provides the bootstrap-your-own-latent framework for images that BYOL-A adapts to audio.","marker":"[29]"},{"why":"The WavLM speech-domain baseline whose out-of-domain transfer performance is contrasted with the SSL models.","marker":"[10]"},{"why":"The AudioSet ontology and CNN architecture behind the Yamnet non-speech baseline used for comparison.","marker":"[25,31]"},{"why":"Introduces representational similarity analysis, the method used to compare model embedding spaces to one another and to acoustic features.","marker":"[38]"},{"why":"Provides the ESC-50 dataset, one of the held-out non-speech sound event recognition transfer tasks.","marker":"[32]"},{"why":"Provides the AVA-Speech dataset, the voice activity detection task that mixes speech and non-speech targets.","marker":"[37]"}],"fun_headline_variants":["Self-supervised audio models shrug off pre-training domain","Pre-training diet doesn't sway self-supervised audio features","Audio SSL: any pre-training domain works nearly as well","Self-supervised audio: domain mismatch barely hurts transfer","Acoustic features from self-supervision are domain-flexible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The clean separation between 'speech' and 'non-speech' pre-training diets assumes the non-speech corpus contains no speech; in fact the MUSAN and AudioSet portions include speech material, so the apparent domain flexibility could partly come from speech leaking into the non-speech model.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised audio models shrug off pre-training domain","Pre-training diet doesn't sway self-supervised audio features","Audio SSL: any pre-training domain works nearly as well","Self-supervised audio: domain mismatch barely hurts transfer","Acoustic features from self-supervision are domain-flexible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1364,"prompt_tokens":966,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":582,"tokens_out":398,"duration_ms":4387,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:24:27.205822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a BYOL-A model on a rigorously speech-free subset of the non-speech corpora (e.g., AudioSet clips with no speech-related labels, MUSAN noise and music only) and a matched speech-only model, then compare their transfer accuracy on a speech task like Speech Commands. If the clean non-speech model loses most of its speech-task performance, the paper's flexibility conclusion would be substantially weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap-your-own-latent framework for images that BYOL-A adapts to audio."},{"cited_title":"Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders,","cited_arxiv_id":null,"evidence_quote":"Introduces representational similarity analysis, the method used to compare model embedding spaces to one another and to acoustic features."}],"review_version":1}