REVIEW 3 major objections 2 minor 2 cited by
A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References
T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Noisy references cap SI-SDR and push noise into separated speech.
desk verdict The abstract's claim that WSJ0-2Mix has noisy references is factually wrong for the standard dataset, which undercuts the paper's central motivation and the interpretation of its correlation result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the SI-SDR itself, a scale-invariant ratio of the energy of the projection of the estimate onto the reference to the energy of the residual. The paper's derivation treats the reference as clean signal plus additive independent noise and solves for the estimate that maximizes the expected SI-SDR, showing that the optimum includes the noise component and that the achievable score is bounded by the reference noise level. The proposed mitigation combines reference enhancement (denoising the references before training) with mixture augmentation using WHAM! noise, so the model learns to reject noise rather than copy it.
What would settle it
Use clean training data, add a measured amount of independent noise to the references, and train a model on the clean references. If its SI-SDR computed against the noisy references comes out clearly above the ceiling predicted by the paper's formula from the reference noise level, the derivation's bound is wrong. A simpler version: find any real benchmark where the same model's score against noisy references exceeds the ceiling computed from the measured reference noise, which would refute the claim that reference noise caps SI-SDR.
Extended reading notes
Core claim
The central claim is that SI-SDR, a projection-based ratio used both to evaluate and to train separation models, is biased by reference noise: when a training reference is a clean source plus additive noise, maximizing SI-SDR pushes a model to reproduce the noise, and the score is capped by how much noise the reference contains. The paper derives this by writing the noisy reference as clean signal plus noise and examining the projection of an estimate onto that reference. To avoid learning the noise, the authors enhance the references and add WHAM! noise to the mixtures during training. Two models trained this way produce separated speech that is less noisy according to NISQA.v2, but the enh
Load-bearing premise
All of the derivation and the proposed fix assume each training reference is a clean speech signal plus additive, independent noise; if real reference noise is correlated with the speech or has structure, the predicted limits and the enhancement recipe may stop working.
Editorial extensions
If this is right
- When reference signals contain noise, SI-SDR is not a clean measure of separation quality: there is a ceiling set by the reference noise, and a model can inflate its score by matching that noise.
- Models trained on noisy references will tend to output noise as part of the separated speech, which the derivation predicts and the experiments observe as increased perceived noisiness.
- Enhancing the training references and augmenting mixtures with WHAM! noise is a concrete recipe that reduces the noisiness of separated speech.
- The recipe is not free: reference processing can introduce artefacts, so practitioners should weigh noise reduction against artefact creation when applying it.
- The negative SI-SDR/noisiness correlation found on WSJ0-2Mix and Libri2Mix means that, under noisy references, a lower SI-SDR should not be automatically interpreted as worse separation.
Reading between the lines
- A direct extension of the derivation is that any projection-based, scale-invariant objective (such as SI-SNR) will inherit the same reference-noise bias, so benchmark designers should control reference quality, not only metric choice.
- The paper's ceiling formula could be turned into a diagnostic: measure the noise level in a benchmark's references, compute the predicted SI-SDR ceiling, and flag any model that appears to exceed it, since that would indicate the model is reproducing noise.
- A testable follow-up is to vary the strength of reference enhancement and measure the artefact penalty, which would give practitioners a stopping rule for how aggressively to denoise references.
- The negative SI-SDR/noisiness correlation suggests a no-reference estimate of noisiness could be derived from SI-SDR alone in noisy-reference benchmarks, although the paper does not make that claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies SI-SDR as a training and evaluation metric for speech separation when the training references are noisy. It claims that noise in references limits achievable SI-SDR or encourages the model to reproduce that noise, and proposes a reference-enhancement method combined with WHAM!-style mixture augmentation. Two models are trained and evaluated with NISQA.v2, and a negative correlation between SI-SDR and perceived noisiness is reported on WSJ0-2Mix and Libri2Mix test sets, interpreted as supporting the derivation. The abstract does not include the derivation, the experimental details, or the numerical results.
Significance. If the derivation is correct and the empirical claims are substantiated, the work would be relevant to the SI-SDR-based training of separation models, especially for datasets with noisy references. However, the central premise about WSJ0-2Mix appears to be factually questionable, which threatens the paper's scope and interpretation. The proposed mitigation (reference enhancement and WHAM! augmentation) could still be a useful practical contribution if validated on a clearly defined noisy-reference setup, but the current abstract does not provide sufficient evidence to assess that.
major comments (3)
- [Abstract (premise)] The abstract asserts that 'the de facto benchmark WSJ0-2Mix' has noisy training references. In the standard, widely used WSJ0-2Mix dataset, each reference is a clean WSJ0 utterance; no additive noise is introduced into the references. If the authors used the standard dataset, this premise is false and the derivation's motivating scenario does not apply. If they used a modified version (e.g., with added WHAM! noise in the references), that version, its noise model (additive, independent, stationary?) and its SNR must be stated explicitly. This is load-bearing: without noisy references, the derived bound and the reference-enhancement motivation are vacuous, and the correlation observed on clean test references cannot be attributed to the proposed mechanism.
- [Derivation (not shown in abstract)] The central claim is a derivation that noise in references limits achievable SI-SDR or leads to undesired noise in separated outputs. The abstract gives no equations, no statement of the noise model (e.g., whether noise is additive, independent of the source, IID, or stationary), and no assumptions about the mixing process. These details are essential to judge whether the result is general or an artifact of a narrow model. Please provide the full derivation with explicit assumptions and discuss its regime of validity, including cases where references are clean.
- [Evaluation/correlation (abstract)] The reported negative correlation between SI-SDR and perceived noisiness is given without numbers, confidence intervals, correlation coefficients, p-values, or the number of models. Furthermore, the correlation is computed on WSJ0-2Mix and Libri2Mix test sets, where references are clean under the standard definition. Even if the training references were noisy, the test-time SI-SDR would be evaluated against clean references, so any correlation would likely be driven by other factors (metric artifacts, model capacity, or training dynamics) rather than by the derived noisy-reference mechanism. The authors need to either compute the correlation on test sets with noisy references or provide an alternative causal chain that explains the observed correlation.
minor comments (2)
- [Abstract (clarity)] The phrase 'as is the case with the de facto benchmark WSJ0-2Mix' should be replaced by an explicit specification of the dataset version and any preprocessing applied to create noisy references. If the authors deliberately added noise to WSJ0-2Mix, this should be described; otherwise, the claim is misleading.
- [Abstract (method)] The proposed enhancement method is described only as 'enhance references.' Provide at least a high-level description of the enhancement procedure (e.g., a trained denoiser, spectral gating, or a separate model) so readers can gauge the potential for introducing artifacts.
Circularity Check
No significant circularity: the derivation is a mathematical consequence of the SI-SDR definition, and the empirical correlation is presented as supporting evidence, not as an input to the derivation.
full rationale
The abstract contains no fitted parameter that is later renamed a prediction, no self-citation chain carrying a load-bearing premise, and no uniqueness or ansatz imported from the authors' prior work. The central 'derivation' is described as revealing that noisy references limit achievable SI-SDR or lead to unwanted noise in separated outputs; this is a consequence of the definition of SI-SDR when the reference contains additive noise, not a conclusion that is assumed in the premise. The proposed mitigation (enhancing references and augmenting with WHAM!) is an application of that derivation, not a circular restatement. The reported negative correlation between SI-SDR and perceived noisiness is offered as empirical support that 'underlines' the derivation; even if this correlation were confounded, it is not itself used as the derivation's input. The abstract's unsupported premise that WSJ0-2Mix references are noisy is a factual/scope concern rather than a circularity, because the argument does not define SI-SDR or the derivation in terms of that dataset's observed behavior. Since no quoted step exhibits a reduction of a claimed result to its own inputs, the appropriate finding is no circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Training references in WSJ0-2Mix are corrupted by additive, independent noise.
- domain assumption NISQA.v2 is a valid non-intrusive proxy for perceived speech quality.
Cite this review
Pith. "Pith review of A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References." pith.science (2026). https://pith.science/paper/DT5KZ6ZI
@misc{pith2026250814623,
author = {Pith},
title = {Pith review of: A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy References},
year = {2026},
howpublished = {\url{https://pith.science/paper/DT5KZ6ZI}},
note = {Machine review of arXiv:2508.14623}
}
read the original abstract
This paper examines the implications of using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) as both evaluation and training objective in supervised speech separation, when the training references contain noise, as is the case with the de facto benchmark WSJ0-2Mix. A derivation of the SI-SDR with noisy references reveals that noise limits the achievable SI-SDR, or leads to undesired noise in the separated outputs. To address this, a method is proposed to enhance references and augment the mixtures with WHAM!, aiming to train models that avoid learning noisy references. Two models trained on these enhanced datasets are evaluated with the non-intrusive NISQA.v2 metric. Results show reduced noise in separated speech but suggest that processing references may introduce artefacts, limiting overall quality gains. Negative correlation is found between SI-SDR and perceived noisiness across models on the WSJ0-2Mix and Libri2Mix test sets, underlining the conclusion from the derivation.
Forward citations
Cited by 2 Pith papers
-
Towards Array-Invariant Speech Enhancement via Geometry-Aware Dynamic Convolution
Microphone coordinates are converted by a small transformer into dynamic convolution weights, letting fixed-array speech-enhancement models process arbitrary array geometries.
-
DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos
DynFOA synthesizes first-order ambisonics from 360 videos by combining 3D Gaussian Splatting scene reconstruction with conditional diffusion modeling and outperforms prior methods on a new 600-clip real-world dataset.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.