{"id":"89ef8a5d-6ea5-42f0-bd0f-6a7d34ed8a39","arxiv_id":"2505.16306","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Layer-wise probing of MusicFM and MuQ shows acoustic-to-semantic feature progression across layers, and single-layer selection often outperforms all-layer aggregation on MIR tasks.","lead":"This paper probes two self-supervised music models, MusicFM and MuQ, layer by layer across 14 music information retrieval tasks, and finds that lower layers encode acoustic details while higher layers encode semantic meaning. The practical takeaway: for a given task, picking the right single layer often beats pooling all layers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Acoustic-to-semantic layer-evolution claim is not backed by variance or significance: single-run probes on small datasets cannot distinguish layer trends from noise.","rationale":"The reader's weakest assumption correctly identifies the missing error bars and repeated-seed stability of the layer-wise probing. I agree that this is the load-bearing weakness, and I would keep the verdict at CONDITIONAL: the raw superiority of SSL features over FBANK+Chroma is plausibly supported, but the headline 'acoustic-to-semantic evolution' claim needs significance evidence and a more direct test of the proposed latent axis. My partial disagreement is that I would place additional weight on the post-hoc task grouping and on the fact that PWCCA only demonstrates decreasing input-similarity, not increasing semantic content. These are fixable with a small additional experiment, so no verdict change beyond the reader's conditional is needed.","tokens_in":7056,"tokens_out":4950,"duration_ms":47163,"concrete_test":"Re-run layer-wise probing for MusicFM and MuQ on the public tasks (GTZAN, Emomusic, VocalSet, NSynth, MTT) with 10 random seeds per layer, fixing train/validation/test splits, and report mean ± std or bootstrap 95% CIs for each entry in Table 1 and Figure 2. Then test the evolution claim directly: for each seed, compute the best-layer index per task and compare the distribution for the paper's acoustic set {Singer, Instrument, Pitch} versus semantic set {Genre, Emotion valence, Emotion arousal, Technique, Structure} with a permutation test. If the median best-layer gap is not significantly greater than 2 layers, or if CIs for adjacent layers overlap for most tasks, the acoustic-to-semantic progression should be softened to 'task-specific layer specialization' rather than a general representational trend.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central second finding in §4.2 — that features \"progressively evolve from Acoustic-level to Semantic-level\" — is inferred from the best-layer column of Table 1 and a visual reading of Figure 2, but all probing numbers are single runs. The gap between the paper's 'acoustic' best layers (MusicFM: Singer=1, Instrument=3, Pitch=4) and its 'semantic' best layers (Genre=5, Emotion=5, Technique=6, Structure=6) is only a few layers, on datasets as small as GTZAN and Emomusic, using a randomly initialized 512-unit probe. No seeds, confidence intervals, or significance tests are reported anywhere, so the difference between layer 4 and layer 5 could easily be probe noise. The PWCCA curve in Figure 3 is more robust but only shows monotone decorrelation from the input; it establishes that input-like acoustic information is lost, not that semantic information increases or that the paper's acoustic-versus-semantic task grouping is the right latent axis. Since some quantitative results use the private MT dataset, the layer pattern cannot be independently rechecked. The SSL-over-baseline headline is plausible, but the paper's interpretive 'acoustic-to-semantic' conclusion is under-supported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical, layer-wise analysis of two large self-supervised music representation models, MusicFM and MuQ, on 14 downstream tasks drawn from the MARBLE benchmark. The authors evaluate per-layer probing accuracies, a learned weighted combination of all layers, and PWCCA similarity between layer embeddings and the model input. They report three main findings: (i) both SSL models outperform a FBANK+Chroma baseline on all tasks, often by large margins; (ii) features appear to evolve from acoustic-level to semantic-level information across layers, with acoustic tasks best served by lower layers and semantic tasks by higher layers; and (iii) for most tasks, selecting a single best layer outperforms using a weighted combination of all layers. The paper also observes that the optimal layer is usually not the final layer, and that the two models show a similar qualitative pattern.","tokens_in":7293,"tokens_out":6648,"duration_ms":49369,"significance":"If the results hold, the paper provides practically useful guidance for MIR practitioners: SSL music models are strong feature extractors, middle layers often suffice, and explicit layer selection can be beneficial. The comprehensive comparison across 14 tasks and two model families is a valuable benchmark contribution, and the consistency of the layer-wise trends across MusicFM and MuQ is suggestive of a general property of this model family. The main limitation is that the central interpretative claim—the acoustic-to-semantic progression—rests on single-run probing numbers without error bars or significance tests, and the private MT dataset is not fully specified. The paper would be strengthened by adding statistical validation and clarifying the layer-selection protocol.","major_comments":[{"comment":"The claim that features 'progressively evolve from Acoustic-level to Semantic-level' is inferred from single-run probing accuracies. For MusicFM, the best layers for acoustic tasks are Singer=1, Pitch=1, and Instrument=3, while for semantic tasks they are Genre=5, Emotion=5, Technique=6, and Structure=6; these differences span only a few layers. No confidence intervals, repeated seeds, or significance tests are reported anywhere in the paper, so the observed ordering of best layers could be within run-to-run noise, especially on the small datasets used (GTZAN, Emomusic, VocalSet). In addition, the PWCCA results in Figure 3 show only monotone decreasing similarity to the input, which supports the loss of input-like acoustic information but does not by itself demonstrate that semantic information increases along the proposed acoustic-semantic axis. Please provide variance estimates or significance tests (e.g., paired tests across layers over multiple probe initializations) and, if possible, a more direct measure of semantic content to support the progression claim.","section":"§4.2, Table 1, Figure 2"},{"comment":"The Sound Quality Classification (SQC) task defines the annotation scale as 0 points for bad quality and 1 point for good quality, but then states that 'scores greater than 1 were classified as Good quality while scores of 1 or lower were classified as Bad quality.' Since the maximum possible average score is 1, this threshold would classify every track as Bad, which is inconsistent with the reported accuracies in Table 1 (e.g., 79.5% for MusicFM and 76.5% for the baseline). Please correct the scale or threshold, and provide details on the number of annotators, inter-annotator agreement, and the actual label distribution for all three MT tasks (MPC, RD, SQC).","section":"§3.4, SQC task"},{"comment":"The layer-scanning results are presented with the 'layer where the best result occurred' in parentheses, but the paper does not specify whether this layer is selected on a held-out validation set or on the test set. If the test set is used for layer selection, the comparison against the weighted-sum method is an oracle comparison, and the conclusion that 'directly selecting a single layer produced better results than using all layers' is not a fair evaluation of a practical layer-selection strategy. Please clarify the selection protocol; if validation-based selection was used, report the corresponding test scores, and if not, re-run the comparison with a validation-based selection rule.","section":"§4.3, Table 1"}],"minor_comments":[{"comment":"There is a typo in 'Self-superviesed learning' in the first sentence of the Introduction; it should be 'Self-supervised learning'.","section":"Introduction"},{"comment":"The token sample rate is written as '25K Hz'; this should be '25 kHz' for clarity.","section":"§3.2"},{"comment":"The dataset name 'V ocalSet' contains an extra space; it should be 'VocalSet'.","section":"§3.4"},{"comment":"The sentence 'The training objective is to predict tokens that have been randomly masked within a sequence. similar to BERT.' is fragmentary; please merge the second fragment into the previous sentence.","section":"§3.1"},{"comment":"The 'Comprehensive tasks' category is defined as tasks with 'both semantic and acoustic labels,' yet Key detection is included in this category without a clear justification; please either reclassify Key detection or explain how it fits the definition.","section":"§3.4"},{"comment":"The red numbers indicating the best-performing layers may be hard to see in grayscale or with color-blind readers; consider adding markers or text labels that do not rely solely on color.","section":"Figure 2"},{"comment":"The statement 'To our knowledge, this is the first work to analyze music SSL models on the layer-wise granularity' is a strong claim; please either provide a more thorough search of the literature (including analyses of MERT and other music SSL models) or soften the claim.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The headline result that SSL models beat low-level features is likely robust given the large margins, but the acoustic-to-semantic progression claim needs stronger statistical support. The private MT dataset and the inconsistency in the SQC task definition are significant reproducibility concerns; I recommend requesting a corrected description of the MT annotation, details on annotator counts and agreement, and ideally release of the dataset or code. The paper would also benefit from reporting multiple seeds and a clear layer-selection protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Layer-wise probing of MusicFM and MuQ is genuinely useful. The paper reports 14 downstream tasks, two large music SSL models, and a consistent pattern: both models beat FBANK+Chroma everywhere, and the best layer for a task usually sits in the middle rather than the final layer. The PWCCA curves add a second line of evidence that lower layers stay closer to the input. As a practical benchmark for people choosing embeddings from these released models, this is a solid contribution.\n\nThe soft spots are concentrated in the interpretation. The headline claim that representations 'progressively evolve from Acoustic-level to Semantic-level' rests on the best-layer column of Table 1 and a visual reading of Figure 2. Those numbers come from single runs with no seeds, confidence intervals, or significance tests. The gap between the acoustic-best layers and semantic-best layers is only a few layers on small datasets like GTZAN and Emomusic, so the ordering could easily be probe noise. PWCCA shows a monotone decrease in similarity to the input, which supports loss of low-level information but does not by itself show that semantic information increases. The private MT dataset with manually set thresholds adds another unreproducible element; the thresholds look ad hoc (0.67, 1.34) and no annotation statistics are given.\n\nThe 'first work to analyze music SSL models on the layer-wise granularity' claim is too strong given MARBLE already does layer-wise evaluation. The real novelty is the specific profiles for MusicFM and MuQ, and that is worth reporting honestly.\n\nNone of this sinks the paper. The SSL-over-baseline result is robust across all 14 tasks, the layer-selection observation is practically useful, and the methodology is standard. The self-citation to MuQ is reasonable because MuQ is the test object. For peer review, I would send it out; the authors should add repeated seeds and error bars, release the MT dataset or describe it more precisely, and soften both the novelty and the acoustic-to-semantic claims. After that, it is a publishable workshop or conference paper.","headline":"Solid layer-wise benchmark for MusicFM and MuQ, but the acoustic-to-semantic evolution claim needs error bars before it can carry weight.","tokens_in":7762,"tokens_out":2413,"would_cite":true,"duration_ms":19876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised music models outperform spectral baselines on every tested task","keywords":["self-supervised learning","music information retrieval","layer-wise analysis","representation learning","MusicFM","MuQ","probing","canonical correlation analysis"],"falsifier":"Run the same 14-task probing on MusicFM and MuQ with, say, five different random seeds per task and layer, and check whether the best-layer rankings stay consistent; if the optimal layer moves by several layers across seeds, the acoustic-to-semantic layer specialization claim is not a stable property of the models.","tokens_in":6878,"feed_emoji":"🎵","tokens_out":4341,"duration_ms":32897,"temperature":0.7,"pith_summary":"This paper asks what information is stored in modern self-supervised music models and how to use it well. By probing two large music representation models, MusicFM and MuQ, across 14 music information retrieval tasks, it argues that these models consistently beat a handcrafted spectral feature baseline. The layer-by-layer results show a steady shift: lower layers carry acoustic, objective information such as pitch and singer identity, while higher layers carry semantic, subjective information such as genre and emotion. The paper further claims that for most tasks the best single layer sits in the middle of the network, not the final layer, and that picking one good layer often beats averaging all layers with learned weights.","feed_headline":"SSL music models beat spectral baselines on all 14 tasks","feed_subtitle":"Layer-by-layer tests show features shift from acoustic to semantic, with the best layers in the middle.","key_machinery":"The analysis rests on three tools. A frozen-feature probing model, a shallow neural network with one 512-unit hidden layer, is trained on features extracted from each encoder layer separately to measure how much task-relevant information each layer holds. PWCCA, a weighted canonical correlation analysis, is used to quantify how similar each layer's representation is to the model's input Mel spectrum. For feature combination, a softmax-weighted sum of all layer outputs is trained alongside the probe, and compared against picking a single best layer.","core_discovery":"The central discovery is a layered organization of learned music representations. Both MusicFM and MuQ outperform FBANK-plus-Chroma on every one of the 14 tested tasks, by margins that are large for semantic tasks such as genre classification and emotion arousal. As activations pass through the encoder, their correlation with the input Mel spectrum decreases steadily, measured by PWCCA, and task-level probing shows that objective acoustic labels are best decoded from lower layers while subjective semantic labels are best decoded from higher layers. This means the networks gradually re-encode acoustic detail into higher-level musical meaning, and the useful information for a particular task ends up concentrated in a middle band of layers, with the exact best layer task-dependent.","pith_inferences":["Layer-picking as a cheap prior: for a new music task, the paper's placement of the optimal layer could be transferred by task type, saving the cost of scanning all layers.","The single-run probing numbers carry no error bars, so small differences between adjacent layers may not be stable; a repeated-seed study could either confirm or dissolve the reported layer rankings.","The same layer-wise probing could be applied to the pre-training objective itself, for example testing whether token targets that are more spectral (Mel-RVQ) versus more abstract (random-projection) push the acoustic-to-semantic transition earlier or later.","A practical extension would be using the best-layer map to prune models: a task needing only acoustic layers could run a truncated forward pass and save computation."],"forward_implications":["For downstream music tasks, extracting features from the best-performing middle layer should outperform using the final layer's output in most cases.","The category of the task gives a rough guide to layer choice: acoustic labels point to lower layers and semantic labels to higher layers.","Since both models show the same trend, the acoustic-to-semantic gradient is likely a shared property of this family of self-supervised music encoders.","Learned weighted sums of all layers rarely beat the best single layer on smaller datasets, suggesting that aggregation can hurt when the probe has little data."],"supporting_citations":[{"why":"Defines MusicFM, the first of the two analyzed foundation models, including its pre-training setup and public checkpoint.","marker":"[2]"},{"why":"Introduces MuQ, the second analyzed model, whose Mel-RVQ token targets differ from MusicFM's random-projection quantizer.","marker":"[10]"},{"why":"Supplies the downstream task suite and evaluation protocol used for the 14 tasks.","marker":"[15]"},{"why":"Provides the BEST-RQ training objective that MusicFM follows, so the comparison isolates token-target differences between the two models.","marker":"[11]"},{"why":"Establishes canonical correlation analysis, the statistical basis for measuring input-layer similarity.","marker":"[19]"},{"why":"Introduces PWCCA, the specific robust variant used to compute layer-wise similarity with the input.","marker":"[22]"},{"why":"Sets the probe paradigm of a shallow downstream model trained on frozen SSL features, which the paper follows.","marker":"[4]"},{"why":"Supplies the learnable weighted-sum method for combining all layers, which the paper compares against single-layer selection.","marker":"[31]"}],"fun_headline_variants":["SSL music models outdo spectral baselines on all 14 tasks","Middle layers hold best music features in SSL models","SSL music models: acoustic lower, semantic higher","Best layers in middle for music SSL tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a single run of probing at each layer gives a stable ranking of how much task-relevant information each layer holds, but it reports no repeated seeds or significance tests, so adjacent-layer differences could be noise.","fun_headline_variants_meta":{"raw":{"variants":["SSL music models outdo spectral baselines on all 14 tasks","Middle layers hold best music features in SSL models","SSL music models: acoustic lower, semantic higher","Best layers in middle for music SSL tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3672,"prompt_tokens":810,"completion_tokens":2862,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":2800}},"tokens_in":426,"tokens_out":2862,"duration_ms":15558,"temperature":1.0,"reasoning_tokens":2800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:02:52.404812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 14-task probing on MusicFM and MuQ with, say, five different random seeds per task and layer, and check whether the best-layer rankings stay consistent; if the optimal layer moves by several layers across seeds, the acoustic-to-semantic layer specialization claim is not a stable property of the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MusicFM, the first of the two analyzed foundation models, including its pre-training setup and public checkpoint."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces MuQ, the second analyzed model, whose Mel-RVQ token targets differ from MusicFM's random-projection quantizer."},{"cited_title":"Freeman, Jessie Wang, Sherry Cai, and KatherineM","cited_arxiv_id":null,"evidence_quote":"Supplies the downstream task suite and evaluation protocol used for the 14 tasks."},{"cited_title":"Self-supervised learning with random-projection quantizer for speech recognition","cited_arxiv_id":null,"evidence_quote":"Establishes canonical correlation analysis, the statistical basis for measuring input-layer similarity."},{"cited_title":"Liu, Cheng- I Lai, Haibin Wu, Jiatong Shi, Xuankai Chang, Hsiang-Sheng Tsai, Wen-Chin Huang, Tzu hsun Feng, Po-Han Chi, Yist Y","cited_arxiv_id":null,"evidence_quote":"Introduces PWCCA, the specific robust variant used to compute layer-wise similarity with the input."},{"cited_title":"However, re- search in this area has been hindered by the limitations in data access [6]","cited_arxiv_id":null,"evidence_quote":"Sets the probe paradigm of a shallow downstream model trained on frozen SSL features, which the paper follows."},{"cited_title":"Neu- ral audio synthesis of musical notes with wavenet autoencoders","cited_arxiv_id":null,"evidence_quote":"Supplies the learnable weighted-sum method for combining all layers, which the paper compares against single-layer selection."}],"review_version":1}