REVIEW 3 major objections 7 minor 26 references
Local Equivariance Error-Based Metrics for Evaluating Sampling-Frequency-Independent Property of Neural Network
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A mask-predictor-focused metric predicts how badly source separation degrades at untrained sampling frequencies.
desk verdict Useful idea and a striking empirical pattern, but the metric is undefined at the point of the Lie derivative because resampling changes the vector dimension and no output transformation is specified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Lie derivative of the network under the resampling transformation: with $h_r(x)=g_r^{-1}(f(g_r(x)))$, LEE is the expected norm of $L_f=(d/dr)h_r|_{r=0}$, and resampling is realized as a matrix $S$ whose entries are windowed sinc kernels depending on the rate $r$. The paper's key move is to apply the chain rule $L f_{NN}=L f_{dec}+J_{dec}L f_{mask}+J_{dec}J_{mask}L f_{enc}$ and define log-normalized metrics (LN-LEE variants) that keep only the mask-predictor term, so the metric reflects the component the authors show to be responsible for degradation.
What would settle it
Train, say, twenty separators with different architectures, losses, or random seeds, and compute Mask-LN-LEE at the trained frequency alongside SDR at several untrained frequencies. If the Spearman correlation across this broader set falls well below the reported $0.9$ or changes sign, then Mask-LN-LEE is not a general proxy for resampling damage. Even a single model with low Mask-LN-LEE and large SDR degradation would weaken the causal reading.
Extended reading notes
Core claim
The paper claims that sampling-frequency independence in a TasNet-style separator is determined by the mask predictor, and that a log-normalized local equivariance error computed on that component quantifies the property. Using windowed-sinc resampling $g_r$ as the input transformation, the authors define three metrics---layerwise LN-LEE, $\Delta$LN-LEE, and Mask-LN-LEE---each isolating the mask predictor's Lie-derivative contribution. Across ten initializations of the SFI Conv-TasNet, Mask-LN-LEE computed at the trained $32$ kHz frequency correlates with SDR degradation at untrained frequencies ($\rho=0.89$ at $8$ kHz and $\rho=0.93$ at $16$ kHz), whereas whole-network LN-LEE is negatively correlated ($\rho=-0.74$ and $-0.83$). The paper therefore establishes that the right component-level metric can predict resampling damage without ever testing at the untrained frequency.
Load-bearing premise
The approach assumes that resampling the signal is a smooth transformation whose derivative with respect to the rate parameter is well-defined and captures the network's response, even though resampling actually changes the length of the signal and therefore does not act as a group on a fixed vector space.
Editorial extensions
If this is right
- Mask-LN-LEE computed at the trained sampling frequency can substitute for measuring SDR at every untrained frequency, saving evaluation cost in model development.
- Whole-network equivariance metrics can be actively misleading for resampling robustness; component-level attribution is necessary.
- If the correlation is causal, lowering Mask-LN-LEE by making the mask predictor more equivariant under resampling should directly improve untrained-frequency separation quality.
- The same LEE decomposition could be applied to other architectures whose trainable time-frequency frontend and backend frame a mask predictor, though the paper only validates TasNet-style models.
- A single trained-frequency measurement may be enough to rank models by their expected resampling damage, which is useful when untrained-frequency test data are unavailable.
Reading between the lines
- A natural extension the paper leaves open is testing whether Mask-LN-LEE tracks degradation for STFT-based separators such as band-split RNNs or TF-GridNet.
- Because the metric is computed at the trained frequency, it could be plugged into a training loop as a differentiable regularizer that penalizes mask-predictor non-equivariance, potentially yielding models that are sampling-frequency-independent by construction.
- The negative whole-network correlation suggests a compensatory effect: the encoder and decoder can appear to 'absorb' resampling while the mask predictor is actually the fragile part, and the same compensation could occur in other equivariant tasks beyond audio.
- The validation set here varies only the initial bandwidth of the latent analog filters, so whether the high correlation persists across different architectures, losses, and datasets is a testable open question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces three metrics—LLN-LEE, ΔLN-LEE, and Mask-LN-LEE—for quantifying the sampling-frequency independence (SFI) of TasNet-style audio source separation networks, based on local equivariance error (LEE) applied to signal resampling. The authors first extend LEE by treating windowed-sinc resampling as an input transformation and define a log-normalized version (LN-LEE); after observing that whole-network LN-LEE correlates negatively with SDR degradation at untrained sampling frequencies, they propose mask-predictor-focused variants. Experiments with SFI Conv-TasNet on MUSDB18-HQ, varying the initial bandwidth parameter σ over ten values, report correlation coefficients ρ between the proposed metrics and SDR degradation at 8 and 16 kHz of roughly 0.83–0.93, while whole-network LN-LEE has ρ≈−0.74 to −0.83.
Significance. The target problem—predicting robustness at untrained sampling frequencies from a metric computed at the training frequency—is timely and practically relevant. The paper is honest about the failure of the naive whole-network LEE and provides a reproducible experimental protocol (public dataset, four random seeds, official LEE implementation). If the proposed metrics are valid, they would give researchers a cheap screening tool for SFI before deploying models across sample rates. However, the mathematical foundation of the extension is incomplete and the empirical evidence is based on ten correlated design points without uncertainty quantification, so the central claim is not yet established at the level required for publication.
major comments (3)
- [Section III-A, Eqs. (6)–(7) and Eq. (3)] The resampling operator S(r) in Eq. (6) maps R^N to R^{ceil(exp(r)N)}, and the output dimension changes with r; for any r>0 it immediately jumps to N+1, so the family {S(r)} is not contained in a single matrix space and the derivative ∂g_r/∂r at r=0 is not well-defined as a two-sided derivative. Equation (3) requires an invertible output representation g'_r and a composition h_r = g'^{-1}_r(f(g_r(x))), but no such g'_r is specified for resampling. Because LN-LEE, LLN-LEE, ΔLN-LEE, and Mask-LN-LEE all inherit Lf from this construction, the metrics are underspecified as written. The authors need to either define a proper dimension-preserving group action (e.g., resampling from N samples to N samples through a variable-rate grid) or replace the Lie-derivative formalism with a finite-difference sensitivity that is well defined across dimensions.
- [Section III-C, Eq. (12)] The approximation used for ΔLN-LEE replaces Lf_dec + J_dec J_mask Lf_enc with Lf_no_mask = Lf_dec + J_dec Lf_enc, i.e., it effectively assumes J_mask Lf_enc ≈ Lf_enc. This is not justified for a nonlinear mask predictor and is not an identity in general. Since ΔLN-LEE is one of the three proposed metrics and is claimed to perform comparably to Mask-LN-LEE, this step needs either a proof, a numerical validation, or an explicit statement that ΔLN-LEE is a heuristic approximation.
- [Section IV-C, Figs. 3–4] The correlations are computed from ten σ-initialization values and each plotted point is the average over four random seeds. No confidence intervals, significance tests, or seed-to-seed variability are reported. With such a small number of design points and several metrics tested at two sampling rates, the reader cannot assess whether the ρ≈0.83–0.93 values and the negative ρ for whole-network LN-LEE are robust. Please report confidence intervals, permutation or bootstrap p-values, and ideally show per-seed scatter points or error bars.
minor comments (7)
- [Section I] The phrase 'howSFI a method is' should read 'how SFI a method is'.
- [Figures 3 and 4] Panel (c) in both figures has the y-axis label 'LN-LEE'; it should read 'ΔLN-LEE' to match the caption.
- [Section III-B] The sentence 'Although the proposed metrics are applicable ... We leave the evaluation ...' should be a single grammatical sentence with a lowercase 'we' after the comma.
- [Section IV-B] Please clarify that the ten plotted points correspond to σ = 10π, 20π, ..., 100π, and describe how the four-seed average was computed before taking the logarithm.
- [Eqs. (10) and (14)] The notation (Lfmask)(fenc(x)) is slightly abusive; please define whether the Lie derivative of f_mask is taken with respect to the mask predictor's input and then evaluated at f_enc(x).
- [Throughout] The acronym 'mask-LN-LEE' appears with a lowercase 'mask' in Eq. (14) while the text and figures use 'Mask-LN-LEE'; please unify the capitalization.
- [Section IV-C] The statement that computing the proposed metrics at the test sampling frequencies gave 'similar trends' is qualitative; report the corresponding correlations or a figure for that comparison.
Circularity Check
No significant circularity: the proposed metrics are defined independently of the SDR degradation target, and the reported correlations are empirical rather than identities.
full rationale
The paper's proposed metrics are defined from the Lie derivative of the network under a resampling transformation (Eqs. 2, 3, 6-8, 10, 13, 14), and none of their definitions contains the SDR degradation values that they are compared against in Figs. 3-4. No parameter in the metric is fitted to the target, and the correlations are empirical outcomes rather than algebraic identities. The only self-citations are to the SFI Conv-TasNet architecture [3] and prior SFI studies [4], [12]; these support the experimental setup and motivation, but they are not used to justify the validity of the proposed metrics. The LEE framework itself is cited from external work [13]. The paper does contain a post-hoc design element: the decision to focus on the mask predictor was made after observing that whole-network LN-LEE behaved contrary to the desired trend (Section III-B), and the same experimental data are then used to report the correlations. This is a validation limitation rather than a circular reduction, because the metric values are not constructed from the SDR degradation target. The mathematical underspecification of the resampling Lie derivative when the output dimension changes in Eq. (6) is a correctness concern, not a circularity concern. Overall, the derivation chain is self-contained with respect to the proposed metrics, and no load-bearing step reduces to its own inputs by construction.
Assumptions & free parameters
free parameters (1)
- initial bandwidth parameter sigma of modulated Gaussian filter (Eq. 15) =
10pi, 20pi, ..., 100pi (10 values)
assumptions (5)
- standard math The Lie derivative chain rule decomposition in Eq. (4) applies to the TasNet composition f_dec o f_mask o f_enc.
- ad hoc to paper Windowed-sinc resampling is a differentiable input transformation with computable derivative dg_r/dr, so LEE extends to it.
- domain assumption The mask predictor is the main source of sampling-frequency-dependent degradation in TasNet.
- ad hoc to paper The approximation Lf_dec + J_dec J_mask Lf_enc being close to Lf_no_mask in Eq. (12) is accurate enough for Delta-LN-LEE.
- domain assumption SDR degradation at 8 and 16 kHz adequately represents the SFI property.
Cite this review
Pith. "Pith review of Local Equivariance Error-Based Metrics for Evaluating Sampling-Frequency-Independent Property of Neural Network." pith.science (2026). https://pith.science/paper/TKIKMVXF
@misc{pith2026250603550,
author = {Pith},
title = {Pith review of: Local Equivariance Error-Based Metrics for Evaluating Sampling-Frequency-Independent Property of Neural Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKIKMVXF}},
note = {Machine review of arXiv:2506.03550}
}
read the original abstract
Audio signal processing methods based on deep neural networks (DNNs) are typically trained only at a single sampling frequency (SF) and therefore require signal resampling to handle untrained SFs. However, recent studies have shown that signal resampling can degrade performance with untrained SFs. This problem has been overlooked because most studies evaluate only the performance at trained SFs. In this paper, to assess the robustness of DNNs to SF changes, which we refer to as the SF-independent (SFI) property, we propose three metrics to quantify the SFI property on the basis of local equivariance error (LEE). LEE measures the robustness of DNNs to input transformations. By using signal resampling as input transformation, we extend LEE to measure the robustness of audio source separation methods to signal resampling. The proposed metrics are constructed to quantify the SFI property in specific network components responsible for predicting time-frequency masks. Experiments on music source separation demonstrated a strong correlation between the proposed metrics and performance degradation at untrained SFs.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep learning for audio signal processing,
H. Purwins, B. Li, T. Virtanen, J. Schl ¨uter, S.-Y . Chang, and T. Sainath, “Deep learning for audio signal processing,”IEEE J. Select. Topics Signal Process., vol. 13, no. 2, pp. 206–219, 2019. 1 2 3 SDR degradation [dB] 0.04 0.06 0.08 0.10LN-LEE 10 20 30 40 50 60 70 80 90 100 1 2 3 SDR degradation [dB] 4.18 4.20 4.22Entire LN-LEE (a) Entire LN-LEE (ρ=−...
work page 2019
-
[2]
Y . C. Eldar,Sampling Theory: Beyond Bandlimited Systems. Cambridge University Press, 2015
work page 2015
-
[3]
Sampling- frequency-independent convolutional layer and its application to audio source separation,
K. Saito, T. Nakamura, K. Yatabe, and H. Saruwatari, “Sampling- frequency-independent convolutional layer and its application to audio source separation,”IEEE/ACM Trans. Audio Speech Lang. Process., vol. 30, pp. 2928–2943, 2022
work page 2022
-
[4]
Algorithms of sampling-frequency-independent layers for non-integer strides,
K. Imamura, T. Nakamura, N. Takamune, K. Yatabe, and H. Saruwatari, “Algorithms of sampling-frequency-independent layers for non-integer strides,” inProc. Eur . Signal Process. Conf., 2023, pp. 326–330
work page 2023
-
[5]
Efficient monaural speech enhancement with uni- versal sample rate band-split RNN,
J. Yu and Y . Luo, “Efficient monaural speech enhancement with uni- versal sample rate band-split RNN,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2023, 5 pages
work page 2023
-
[6]
Toward universal speech enhancement for diverse input conditions,
W. Zhang, K. Saijo, Z.-Q. Wang, S. Watanabe, and Y . Qian, “Toward universal speech enhancement for diverse input conditions,” inProc. Autom. Speech Recognit. Underst. Workshop, 2023, 6 pages
work page 2023
-
[7]
URGENT challenge: Universality, robustness, and generalizability for speech enhancement,
W. Zhang, R. Scheibler, K. Saijo, S. Cornell, C. Li, Z. Ni, J. Pirklbauer, M. Sach, S. Watanabe, T. Fingscheidtet al., “URGENT challenge: Universality, robustness, and generalizability for speech enhancement,” inProc. INTERSPEECH, 2024, pp. 4868–4872
work page 2024
-
[8]
Performance measurement in blind audio source separation,
E. Vincent, R. Gribonval, and C. Fevotte, “Performance measurement in blind audio source separation,”IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 4, pp. 1462–1469, 2006
work page 2006
Show all 26 references
-
[9]
SDR – half- baked or well done?
J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR – half- baked or well done?” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2019, pp. 626–630
2019
-
[10]
30+ years of source separation research: Achievements and future challenges,
S. Araki, N. Ito, R. Haeb-Umbach, G. Wichern, Z.-Q. Wang, and Y . Mitsufuji, “30+ years of source separation research: Achievements and future challenges,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2025, 5 pages
2025
-
[11]
Sampling frequency independent dialogue separation,
J. Paulus and M. Torcoli, “Sampling frequency independent dialogue separation,” inProc. Eur . Signal Process. Conf., 2022, pp. 160–164
2022
-
[12]
Neural analog filter for sampling-frequency-independent convolutional layer,
K. Imamura, T. Nakamura, K. Yatabe, and H. Saruwatari, “Neural analog filter for sampling-frequency-independent convolutional layer,”APSIPA Trans. Signal Inf. Process., vol. 13, no. 1, 2024
2024
-
[13]
The Lie derivative for measuring learned equivariance,
N. Gruver, M. A. Finzi, M. Goldblum, and A. G. Wilson, “The Lie derivative for measuring learned equivariance,” inProc. Int. Conf. Learn. Represent., 2023, 9 pages
2023
-
[14]
Imaging with equivariant deep learning: From unrolled network design to fully unsupervised learning,
D. Chen, M. Davies, M. J. Ehrhardt, C.-B. Sch ¨onlieb, F. Sherry, and J. Tachella, “Imaging with equivariant deep learning: From unrolled network design to fully unsupervised learning,”IEEE Signal Process. Mag., vol. 40, no. 1, pp. 134–147, 2023
2023
-
[15]
TasNet: Time-domain audio separation network for real-time, single-channel speech separation,
Y . Luo and N. Mesgarani, “TasNet: Time-domain audio separation network for real-time, single-channel speech separation,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2018, pp. 696–700
2018
-
[16]
Conv-TasNet: Surpassing ideal time-frequency magnitude mask- ing for speech separation,
——, “Conv-TasNet: Surpassing ideal time-frequency magnitude mask- ing for speech separation,”IEEE/ACM Trans. Audio Speech Lang. Process., vol. 27, no. 8, pp. 1256–1266, 2019
2019
-
[17]
Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation,
Y . Luo, Z. Chen, and T. Yoshioka, “Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2020, pp. 46– 50
2020
-
[18]
Music source separation with band-split RNN,
Y . Luo and J. Yu, “Music source separation with band-split RNN,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 31, pp. 1893–1901, 2023
1901
-
[19]
Music source separation with band-split RoPE transformer,
W.-T. Lu, J.-C. Wang, Q. Kong, and Y .-N. Hung, “Music source separation with band-split RoPE transformer,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2024, pp. 481–485
2024
-
[20]
TF-GridNet: Integrating full-and sub-band modeling for speech sep- aration,
Z.-Q. Wang, S. Cornell, S. Choi, Y . Lee, B.-Y . Kim, and S. Watanabe, “TF-GridNet: Integrating full-and sub-band modeling for speech sep- aration,”IEEE/ACM Trans. Audio Speech Lang. Process., vol. 31, pp. 3221–3236, 2023
2023
-
[21]
MUSDB18-HQ - an uncompressed version of MUSDB18,
Z. Rafii, A. Liutkus, F.-R. St ¨oter, S. I. Mimilakis, and R. Bittner, “MUSDB18-HQ - an uncompressed version of MUSDB18,” 2019
2019
-
[22]
Meta-learning extractors for music source separation,
D. Samuel, A. Ganeshan, and J. Naradowsky, “Meta-learning extractors for music source separation,” inProc. IEEE Int. Conf. Acoust. Speech Signal Process., 2020, pp. 816–820
2020
-
[23]
On the variance of the adaptive learning rate and beyond,
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” inProc. Int. Conf. Learn. Represent., 2020, 13 pages
2020
-
[24]
Lookahead optimizer: ksteps forward, 1 step back,
M. Zhang, J. Lucas, J. Ba, and G. E. Hinton, “Lookahead optimizer: ksteps forward, 1 step back,” inProc. Adv. Neural Inf. Process. Syst., vol. 32, 2019, pp. 9597–9608
2019
-
[25]
Single- channel multi-speaker separation using deep clustering,
Y . Isik, J. Le Roux, Z. Chen, S. Watanabe, and J. R. Hershey, “Single- channel multi-speaker separation using deep clustering,” inProc. IN- TERSPEECH, vol. 8, 2016, pp. 545–549
2016
-
[26]
The 2018 signal separation evaluation campaign,
F.-R. St ¨oter, A. Liutkus, and N. Ito, “The 2018 signal separation evaluation campaign,” inProc. Int. Conf. Latent V ar . Anal. Signal Sep., 2018, pp. 293–305
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.