{"id":"92adb11a-8288-45dc-ba5b-6af0ba68a6be","arxiv_id":"2607.06392","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"InsideSSL analyzes self-supervised speech models layer-by-layer using entropy, curvature, robustness metrics, and a cross-layer Generative Compatibility Matrix, finding that training objectives induce distinct compression and geometric regimes that correlate with downstream task performance.","lead":"This paper introduces InsideSSL, a framework for analyzing how self-supervised speech models (like Wav2Vec2, HuBERT, and WavLM) encode information layer by layer. A smart generalist might read it to understand which layers of these models are best for specific tasks like recognizing phonemes or speakers, and how training objectives shape internal representations.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The von Neumann entropy of the Gram matrix (Eq. 1) measures spectral diversity (effective rank), not information content; the paper's interpretation of entropy decrease as beneficial 'compression' rather than destructive 'collapse' is unvalidated and load-bearing for the central claim.","rationale":"The reader correctly identified the proxy validation issue as the weakest assumption. I agree this is the most load-bearing concern. My contribution is to make it more precise: the concern is not merely that the proxies are 'unvalidated for speech' but that the von Neumann entropy of the Gram matrix is a spectral diversity measure, not an information-theoretic quantity, creating a category error in the interpretation. The paper's own probing data (non-monotonic curves, Wav2Vec2's positive entropy-phoneme correlation) provides internal evidence that the 'compression = beneficial' interpretation may not hold uniformly. The verdict remains CONDITIONAL because the paper's descriptive contributions (documenting layer-wise trajectories, introducing the GCM) have value regardless of the interpretive framing, but the causal/interpretive claims about compression benefiting phoneme recognition are not adequately supported. The paper itself acknowledges this gap in its conclusion ('future work must establish formal causal links'). The GCM methodology is a reasonable novel contribution and does not share the same proxy concern since it directly tests functional compatibility through reconstruction. Statistical rigor (error bars, significance tests on correlations computed over ~12 layers) would strengthen the paper but is secondary to the fundamental interpretive gap.","tokens_in":14081,"tokens_out":5467,"duration_ms":389023,"concrete_test":"For Wav2Vec2-base, compute a nonlinear mutual information estimate (e.g., via a sufficiently expressive probe such as a 2-layer MLP, or a kernel-based MI estimator) between each layer's representations and phoneme labels, focusing on layers 9–12 where the entropy collapse occurs. If MI(Z^(11); Y_phoneme) < MI(Z^(9); Y_phoneme), the entropy collapse is destructive (information loss), not beneficial compression, and the claim that 'phoneme recognition benefits from deep-layer compression' would need to be qualified as model-dependent rather than general.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central take-away — 'phoneme recognition benefits from deep-layer compression and linearization' — depends on interpreting the von Neumann entropy of K^(ℓ) = Z^(ℓ)(Z^(ℓ))^T (Eq. 1) as a measure of 'informational density.' However, this entropy quantifies the spectral spread (effective rank) of the embedding matrix, which is not equivalent to information content in the Information Bottleneck sense (e.g., I(Z;X) or I(Z;Y)). A decrease in spectral diversity can indicate either (a) beneficial compression that removes noise while preserving task-relevant signal, or (b) destructive collapse that discards information. The paper does not distinguish these cases. This is load-bearing because the claim that compression 'benefits' phoneme recognition requires interpretation (a), but no independent measure of task-relevant information is provided to confirm it. In fact, the linear probing curves (Figure 8a) are non-monotonic — phoneme accuracy peaks in mid-layers and often declines in the deepest layers — which is more consistent with over-compression (information loss) in terminal layers than with beneficial compression. The Pearson correlations in Figure 9 assume a linear relationship between entropy and task performance, but the probing curves' non-monotonicity means these correlations may mischaracterize the relationship. Notably, Wav2Vec2 — the model with the most dramatic entropy collapse — shows a positive entropy-phoneme correlation (0.33 in Figure 9a), opposite to the averaged trend (-0.46), suggesting the 'compression helps phonemes' claim does not generalize to the model that most clearly exhibits the compression phenomenon.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes InsideSSL, a model-centric framework for analyzing self-supervised speech representation (SSL) models across their Transformer layers. The framework has two components: (1) three per-layer intrinsic metrics—von Neumann entropy of the Gram matrix (compression), average curvature of token transitions (geometry), and InfoNCE-based invariance to perturbations (robustness)—and (2) a cross-layer Generative Compatibility Matrix (GCM) that trains per-layer CFM+DiT decoders and evaluates their cross-layer transferability. The authors apply these tools to Wav2Vec2, HuBERT, WavLM, Data2Vec, and UniSpeech at BASE/PLUS/LARGE scales, and connect the intrinsic metrics to downstream phoneme, pitch, and speaker probing. The central claims are that training objectives induce distinct compression and manifold-unfolding regimes, and that phoneme recognition benefits from deep-layer compression/linearization while pitch and speaker tasks rely on early high-entropy, high-curvature states.","tokens_in":15046,"tokens_out":1357,"duration_ms":313289,"significance":"The paper provides a systematic and broad-coverage empirical study: five model families, three scales, fine-tuning data ablations, training-dynamics tracking, and a novel cross-layer GCM methodology. The GCM is a genuinely new contribution for audio SSL, moving beyond isolated per-layer probes to quantify inter-layer functional relationships. The provision of a project page with code and interactive audio is a positive for reproducibility. The connection of intrinsic, task-agnostic metrics to downstream probing via Pearson correlations is a reasonable bridging strategy. The paper's scope and the novelty of the GCM make it a solid contribution to the SSL interpretability literature.","major_comments":[{"comment":"§2.2 (Compression, Eq. 1) and §3.5: The paper interprets von Neumann entropy of the Gram matrix as 'informational density' and frames its decrease as beneficial 'compression.' However, this entropy measures spectral spread (effective rank), not information content in the Information Bottleneck sense (I(Z;X) or I(Z;Y)). A decrease in effective rank can indicate either beneficial compression (noise removal with signal preservation) or destructive collapse (loss of task-relevant information). The paper does not distinguish these cases. This is load-bearing for the central claim that 'phoneme recognition benefits from deep-layer compression and linearization.' The authors should either (a) add an independent measure of task-relevant information (e.g., mutual information estimates or probing-based proxies) to validate that the entropy decrease preserves phoneme-relevant signal, or (b) soften'","section":null},{"comment":"Figure 9a, Wav2Vec2 row: The entropy-phoneme correlation for Wav2Vec2 is +0.33, which is positive—meaning higher entropy is associated with better phoneme accuracy. This is opposite to the paper's general claim that phoneme recognition benefits from low entropy (compression). The paper's take-away states phonemes require 'deep-layer compression,' but Wav2Vec2—the model with the most dramatic entropy collapse—shows the opposite trend. This inconsistency should be explicitly addressed. If the claim is model-dependent, the take-away should be qualified accordingly.","section":null},{"comment":"Figure 9 and §3.5: The Pearson correlations between intrinsic metrics and probing accuracy assume a linear relationship. However, Figure 8a shows that phoneme accuracy is non-monotonic (peaking in mid-layers and declining in deep layers for most models). A linear correlation coefficient may mischaracterize a non-monotonic relationship. The authors should discuss this limitation and consider whether rank-based or non-monotonic measures would be more appropriate for the reported correlations.","section":null},{"comment":"§3.1–§3.5: No error bars, confidence intervals, or significance tests are reported for any of the intrinsic metrics, GCM entries, or probing results. Given that the central claims rest on comparing layer-wise trajectories and correlation values across models, some quantification of variance is needed to assess whether the observed differences (e.g., Wav2Vec2's entropy collapse vs. WavLM's stability) are statistically reliable. At minimum, bootstrap confidence intervals on the entropy and curvature averages, or standard deviations across the test set, would strengthen the reported patterns.","section":null}],"minor_comments":[{"comment":"§2.2, Eq. (2): The curvature formula uses v_n^(l) but the text defines l as the layer index elsewhere. The superscript notation switches between (l) and (ℓ); consistent notation would help.","section":null},{"comment":"§2.3: The GCM definition uses M(D^(ℓ)(Z^(k)), x) but does not specify what x is in this context (waveform? Mel-spectrogram?) at the point of definition. This is clarified later in §3.4 but should be stated at the point of definition.","section":null},{"comment":"Table 1: The 'Task' column uses P/C/D abbreviations but UniSpeech is marked as both P and C. A brief note on UniSpeech's dual objective would help readers unfamiliar with this model.","section":null},{"comment":"Figure 2: The y-axis label 'Average Normalized Entropy' in Fig. 2a does not specify the normalization scheme. The text mentions maxEntropy normalization in §3.1, but the figure caption should clarify this.","section":null},{"comment":"§3.3, Figure 6: The caption says 'first training iteration of a HuBERT model, utilizing labels extracted from MFCCs.' It is unclear whether this refers to the first iteration of pre-training or the first iteration of a specific training stage. Clarification is needed.","section":null},{"comment":"§3.4: The GCM is described as 'structurally asymmetric, strongly favoring the lower triangular region.' The text should clarify the matrix convention (row = training layer, column = evaluation layer) at this point to make the asymmetry claim more precise.","section":null},{"comment":"References [18] and [19] appear to be the same arXiv preprint (arXiv:2501.05310) cited twice with slightly different author lists. This should be corrected.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the entropy metric measuring effective rank rather than information content is valid and is the primary reason for the major revision recommendation. The Wav2Vec2 positive entropy-phoneme correlation is a concrete internal inconsistency that the authors need to address. The GCM methodology is novel and sound; the paper's main weakness is the interpretive framing of the entropy metric rather than any methodological flaw."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the thorough and constructive report. The referee correctly identifies the GCM as a novel contribution and acknowledges the breadth of our study. The four major comments raise substantive points about (1) the interpretation of von Neumann entropy as 'compression,' (2) the Wav2Vec2 sign inconsistency in entropy-phoneme correlation, (3) the appropriateness of linear Pearson correlation for non-monotonic relationships, and (4) the absence of error bars and significance tests. We address each below and commit to revisions for all four points.","responses":[{"response":"The referee is correct that von Neumann entropy of the Gram matrix measures spectral spread (effective rank) rather than mutual information in the Information Bottleneck sense, and that a decrease in effective rank is ambiguous between beneficial compression and destructive collapse. We agree this distinction is important and that our current framing overstates what the entropy metric alone can establish. We will revise the manuscript in two ways. First, we will soften the language throughout: 'informational density' will be replaced with 'effective dimensionality' or 'spectral spread,' and we will explicitly state that entropy decrease is consistent with—but does not prove—beneficial compression. Second, we will note that our linear probing results (Section 3.5, Figure 8a) serve as an independent, task-relevant proxy: phoneme accuracy does not catastrophically decline in the deep layers for HuBERT, WavLM, and UniSpeech, which is consistent with beneficial rather than destructive compression for those models. However, we acknowledge that for Wav2Vec2, the entropy collapse coincides with a decline in phoneme accuracy in the final layers, which may indeed reflect partial destructive collapse. We will state this explicitly. We will not claim that entropy alone validates beneficial compression.","revision_made":"yes","referee_comment":"§2.2 (Compression, Eq. 1) and §3.5: The paper interprets von Neumann entropy of the Gram matrix as 'informational density' and frames its decrease as beneficial 'compression.' However, this entropy measures spectral spread (effective rank), not information content in the Information Bottleneck sense (I(Z;X) or I(Z;Y)). A decrease in effective rank can indicate either beneficial compression (noise removal with signal preservation) or destructive collapse (loss of task-relevant information). The paper does not distinguish these cases. This is load-bearing for the central claim that 'phoneme recognition benefits from deep-layer compression and linearization.' The authors should either (a) add an independent measure of task-relevant information (e.g., mutual information estimates or probing-based proxies) to validate that the entropy decrease preserves phoneme-relevant signal, or (b) soften."},{"response":"The referee has identified a genuine inconsistency that we had not adequately addressed. The +0.33 correlation for Wav2Vec2 is indeed opposite in sign to the negative average correlation (-0.46) and to the trends observed for HuBERT (-0.82), Data2Vec (-0.74), and UniSpeech (-0.72). This is not a minor discrepancy: Wav2Vec2 is the model with the most dramatic entropy collapse, and it is the one model where higher entropy is associated with better phoneme accuracy. We agree that the take-away should be qualified. In the revised manuscript, we will explicitly discuss this Wav2Vec2 exception, noting that the entropy collapse in Wav2Vec2's final layers may represent destructive rather than beneficial compression—consistent with the concurrent decline in phoneme accuracy and the GCM's semantic rupture at layer 11. The general claim will be restated as model-dependent: for masked-prediction models (HuBERT, WavLM, UniSpeech), phoneme accuracy correlates negatively with entropy, but for Wav2Vec2's contrastive objective, the relationship reverses. The take-away will be revised to reflect this qualification.","revision_made":"yes","referee_comment":"Figure 9a, Wav2Vec2 row: The entropy-phoneme correlation for Wav2Vec2 is +0.33, which is positive—meaning higher entropy is associated with better phoneme accuracy. This is opposite to the paper's general claim that phoneme recognition benefits from low entropy (compression). The paper's take-away states phonemes require 'deep-layer compression,' but Wav2Vec2—the model with the most dramatic entropy collapse—shows the opposite trend. This inconsistency should be explicitly addressed. If the claim is model-dependent, the take-away should be qualified accordingly."},{"response":"This is a valid methodological concern. The phoneme accuracy curves in Figure 8a are indeed non-monotonic—peaking in mid-layers and declining in deep layers—so Pearson correlation, which captures only linear association, can mischaracterize the relationship between entropy/curvature and phoneme accuracy. We will address this in two ways. First, we will add an explicit discussion of this limitation in Section 3.5, noting that Pearson correlations may understate or misrepresent non-monotonic relationships. Second, we will compute Spearman rank correlations as a supplementary measure and report them alongside the Pearson values. We expect that rank-based measures will better capture the monotonic component of the relationship (e.g., the general trend that lower entropy layers tend to have higher phoneme accuracy for masked-prediction models), while also being more honest about the non-monotonic structure. If the Spearman correlations differ substantially from the Pearson values, we will report both and discuss the discrepancy.","revision_made":"yes","referee_comment":"Figure 9 and §3.5: The Pearson correlations between intrinsic metrics and probing accuracy assume a linear relationship. However, Figure 8a shows that phoneme accuracy is non-monotonic (peaking in mid-layers and declining in deep layers for most models). A linear correlation coefficient may mischaracterize a non-monotonic relationship. The authors should discuss this limitation and consider whether rank-based or non-monotonic measures would be more appropriate for the reported correlations."},{"response":"The referee is right that the absence of variance estimates is a weakness, particularly given that the central claims rest on comparing layer-wise trajectories across models. We will add bootstrap confidence intervals for the intrinsic metrics (entropy, curvature, InfoNCE) computed over the test-clean set. For the GCM entries, we will report standard deviations across the evaluation set. For the linear probing results, we will report standard deviations across multiple random seeds for probe training. We note that for the intrinsic metrics, the patterns we report (e.g., Wav2Vec2's entropy collapse, the curvature transition point) are visually dramatic and consistent across the 2,620 utterances in test-clean, so we expect the confidence intervals to be narrow and the patterns to remain significant. However, we agree that this should be verified empirically rather than assumed, and we will include the variance estimates in the revised figures.","revision_made":"yes","referee_comment":"§3.1–§3.5: No error bars, confidence intervals, or significance tests are reported for any of the intrinsic metrics, GCM entries, or probing results. Given that the central claims rest on comparing layer-wise trajectories and correlation values across models, some quantification of variance is needed to assess whether the observed differences (e.g., Wav2Vec2's entropy collapse vs. WavLM's stability) are statistically reliable. At minimum, bootstrap confidence intervals on the entropy and curvature averages, or standard deviations across the test set, would strengthen the reported patterns."}],"tokens_in":14115,"tokens_out":1593,"duration_ms":164164,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things to know: (1) the Generative Compatibility Matrix (GCM) is a genuinely new cross-layer analysis tool for speech SSL, and the per-layer metrics adapted from the language model information-flow framework are applied systematically across five model families. (2) The central interpretive claim — that entropy decrease represents beneficial compression — is not independently validated, and the paper's own probing data partially contradicts it in the deepest layers. The stress-test concern about von Neumann entropy measuring effective rank rather than information content is correct and does land on reading the paper. The entropy of the Gram matrix captures spectral diversity, not mutual information in the Information Bottleneck sense. A decrease can mean beneficial compression or destructive collapse, and the paper does not distinguish these. This matters because the headline takeaway — 'phoneme recognition benefits from deep-layer compression' — requires the beneficial interpretation. But Figure 8a shows phoneme accuracy peaks in mid-layers and declines in terminal layers for most models, which is more consistent with over-compression than beneficial compression. The paper actually acknowledges this in its discussion of 'over-compress the signal' at deeper layers, but the framing in the take-away box and conclusion overstates the case. The Wav2Vec2 counterexample is telling: it has the most dramatic entropy collapse yet shows a positive entropy-phoneme correlation (0.33), opposite to the averaged trend (-0.46). That said, the GCM is real and useful. Training per-layer CFM+DiT decoders and cross-evaluating them is a clean experimental design, and the asymmetric lower-triangular structure revealing hierarchical pruning is a concrete finding. The curvature analysis across training dynamics (Figure 6) and the fine-tuning scale experiments (Figure 4) add genuine value. The inter-model correlation matrices showing HuBERT/WavLM/UniSpeech clustering together while Wav2Vec2 and Data2Vec form a separate group is a concrete, checkable result. The lack of error bars on any metric curves is a real gap — we cannot assess whether the differences between models are statistically meaningful. The Wav2Vec2 entropy collapse attribution to quantization is explicitly hedged by the authors themselves, which is honest but leaves the question open. This paper is for speech SSL practitioners who want diagnostic tools for layer selection and model comparison. The GCM and the systematic per-layer metric suite earn a serious referee. The entropy interpretation needs to be either softened to match what the metric actually measures (spectral diversity / effective rank) or backed by an independent information-theoretic measure. Recommend accept for peer review with a revision targeting the interpretive framing.","headline":"Solid diagnostic toolkit for speech SSL with one load-bearing interpretive gap","tokens_in":14895,"tokens_out":606,"would_cite":false,"duration_ms":94346,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Speech AI's internal layers follow a predictable geometry","keywords":[],"falsifier":"If the entropy and curvature metrics were replaced with alternative measures of compression and geometry, the correlations with downstream task performance (Figure 9) would likely weaken or vanish.","tokens_in":14172,"feed_emoji":"🔊","tokens_out":843,"duration_ms":116121,"temperature":0.7,"pith_summary":"The paper introduces InsideSSL, a framework for understanding self-supervised speech models (like Wav2Vec2, HuBERT, and WavLM) by examining their internal layer-by-layer structure rather than just their performance on tasks. The authors track three properties across network depth: entropy (how compressed the information is), curvature (how smooth or tangled the representation manifold is), and robustness (how stable the representations are to audio perturbations). They find that different training objectives create distinct internal regimes. For instance, Wav2Vec2 undergoes a sharp entropy collapse in its final layers, while HuBERT and WavLM maintain steady information density throughout. The paper also introduces the Generative Compatibility Matrix (GCM), which measures how well a decoder trained on one layer can interpret representations from another layer, revealing that phonetic content forms a stable core in mid-layers while speaker identity is more volatile. By connecting these intrinsic geometric properties to linear probing results for phoneme, pitch, and speaker tasks, the authors demonstrate that low-level acoustic tasks (pitch, speaker identity) depend on early layers with high entropy and high curvature, while phoneme recognition benefits from deeper layers where the manifold has been compressed and linearized.","feed_headline":"Speech models compress and unfold predictably across layers","feed_subtitle":"Different training objectives create distinct internal geometries that dictate where phonemes, pitch, and speaker identity are best encoded.","key_machinery":"von Neumann entropy of the Gram matrix (measuring informational density), average curvature of token transition vectors (measuring manifold smoothness), InfoNCE-based robustness metric, and the Generative Compatibility Matrix (cross-layer decoder transferability).","core_discovery":"The central finding is that the internal geometry of self-supervised speech models follows a predictable trajectory—early layers build complex, high-dimensional representations (high entropy, high curvature), while deeper layers compress and linearize them—and that this trajectory directly dictates which layers are optimal for different downstream tasks. Phoneme recognition peaks at mid-to-deep layers where curvature transitions from high to low (marking a shift from local acoustic detail to linearly separable abstractions), while pitch and speaker identity rely on the high-entropy, high-curvature states of early layers. The paper also shows that training objectives create distinct regimes:W","pith_inferences":["If entropy and curvature are indeed faithful proxies for informational density and manifold linearization, then one could predict optimal probing layers for novel tasks without running expensive probing experiments, simply by inspecting the curvature transition point.","The finding that scaling model size (WavLM-LARGE) has a stronger structural effect than increasing training data suggests that architectural capacity, not data volume, is the primary driver of internal representational geometry in these models.","The asymmetry of the GCM (deep-layer decoders generalize to early layers but not vice versa) implies a one-way information bottleneck that could constrain the design of multi-layer feature fusion methods."],"forward_implications":["Speech model architectures could be designed with task-specific layer extraction points rather than defaulting to the final layer, improving efficiency for applications like speaker verification (early layers) vs. speech recognition (mid layers).","Pre-training objectives could be evaluated by their effect on internal geometry (entropy stability, curvature linearization) rather than only downstream task scores, potentially guiding the design of objectives that avoid undesirable phenomena like Wav2Vec2's late-stage entropy collapse.","The Generative Compatibility Matrix methodology could be applied to other modalities (vision, language) to map cross-layer functional dependencies in foundation models."],"fun_headline_variants":["Speech SSL models follow predictable layer-wise compression trajectories","Layer geometry dictates phoneme pitch and speaker encoding in speech models","Training objectives shape distinct internal geometries in speech SSL models","Early layers encode identity while deeper layers prune to phonetic abstractions","Cross-layer analysis reveals stable phonetic cores in self-supervised speech models"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that von Neumann entropy of the Gram matrix faithfully measures 'informational density' and that average curvature of token transitions faithfully measures 'manifold unfolding' in the speech domain, without independently validating that these mathematical proxies correspond to the semantic phenomena they are claimed to capture.","fun_headline_variants_meta":{"raw":{"variants":["Speech SSL models follow predictable layer-wise compression trajectories","Layer geometry dictates phoneme pitch and speaker encoding in speech models","Training objectives shape distinct internal geometries in speech SSL models","Early layers encode identity while deeper layers prune to phonetic abstractions","Cross-layer analysis reveals stable phonetic cores in self-supervised speech models"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":571,"prompt_tokens":503,"completion_tokens":68,"prompt_tokens_details":null},"tokens_in":503,"tokens_out":68,"duration_ms":12108,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T07:13:56.451070+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the entropy and curvature metrics were replaced with alternative measures of compression and geometry, the correlations with downstream task performance (Figure 9) would likely weaken or vanish.","supporting_citations":[],"review_version":1}