REVIEW 4 major objections 5 minor 1 cited by
DeepFilterGAN: A Full-band Real-time Speech Enhancement System with GAN-based Stochastic Regeneration
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read DeepFilterGAN claims that a full-band, real-time speech enhancer can improve perceived quality by regenerating speech from a two-channel input of noisy and intermediate-enhanced spectrograms.
desk verdict A competent engineering paper with a clean noisy-conditioning ablation, but the headline MOS gain is measured against a weakened retrained first stage, so the broader claim about improving a strong predictive enhancer is not yet supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Stochastic regeneration with two-channel conditioning. The first stage produces an intermediate enhanced spectrogram $Z(k,f)$; the generator receives $[Y(k,f), Z(k,f)]$ as a two-channel input and is trained to sample the clean speech distribution $P(x|y,z)$ under a multi-scale adversarial loss. The generator is a lightweight modification of Online SpatialNet (1.14M parameters) and the discriminator follows MelGAN's three-scale design; the final objective is $\min_G \mathbb{E}_{y,z}[\sum_l -D_l(\hat{x})] + \beta\|x-\hat{x}\|_1$, where $\hat{x} = \operatorname{istft}(G(y,z))$.
What would settle it
Run the two variants on test utterances where the first stage removes entire phonemes; if the no-concat variant still recovers those phonemes, or if replacing the noisy channel with a fixed constant keeps NISQA-MOS near 3.12, the paper's noisy-conditioning mechanism is not what carries the gain.
Extended reading notes
Core claim
The discovery is that stochastic regeneration with a GAN can repair over-suppression in a real-time predictive enhancer. Specifically, the generator receives the concatenated spectrograms $[Y(k,f), Z(k,f)]$ and is trained against three MelGAN-style discriminators with a hinge loss plus a time-domain L1 term, learning the conditional distribution $P(x|y,z)$. On the challenge's non-blind test set, the final system scores 3.12 NISQA-MOS versus 2.66 for the retrained first stage, holds PESQ, ESTOI, SDR, and LSD roughly constant, and obtains the best overall ranking among compared models. A spectrogram example shows a speech segment removed by the first stage partially reappearing only when the noisy input is included.
Load-bearing premise
The load-bearing premise is that a 1.14M-parameter generator can learn the clean-speech distribution from a two-channel input of noisy and first-stage-enhanced spectrograms, so that the measured NISQA-MOS gain is genuine restoration rather than a generic reshaping of the output.
Editorial extensions
If this is right
- The full system improves NISQA-MOS from 2.66 to 3.12 over the retrained first stage on the 2024 URGENT non-blind test set, while other objective metrics stay roughly level.
- Concatenating the noisy input with the first-stage output is load-bearing for the improvement: removing it lowers NISQA-MOS to 2.86 and severely reduces recovery of over-suppressed speech content.
- A generative second stage can fit inside a real-time system: 3.58M training parameters (3.45M at inference) and 40 ms algorithmic latency.
- Despite lower NISQA-MOS than a larger diffusion-based comparison system, DeepFilterGAN obtains the best overall ranking across nonintrusive, intrusive, downstream-task-independent, and downstream-task-dependent metric categories.
- The system preserves intelligibility-side metrics (phoneme similarity and word accuracy rate) better than the larger diffusion-based comparison.
Reading between the lines
- The noisy conditioning channel may act as more than noise context: it could anchor the generator to the input signal, and a scrambling experiment could separate those two roles.
- Jointly training both stages, which the paper lists as future work, is a natural next step because a frozen first stage may limit how well the generator can learn $P(x|y,z)$; end-to-end training could widen the NISQA-MOS gap.
- Because the second stage has only 1.14M parameters, the same noisy-concat conditioning could be tested on even smaller on-device models.
- The time-domain L1 term in the generator objective probably explains why intrusive metrics stay stable; varying $\beta$ would directly probe the trade-off between adversarial sharpness and signal conservation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DeepFilterGAN, a two-stage causal speech enhancement system. The first stage is a retrained DeepFilterNet2; the second stage is a GAN whose generator is based on OnlineSpatialNet and whose discriminator is based on MelGAN. The generator receives the concatenation of the noisy and intermediate-enhanced STFTs, which the authors call noisy conditioning. The system has 3.58M parameters and 40 ms algorithmic latency. Experiments on the 2024 URGENT non-blind test set report NISQA-MOS of 3.12 for the proposed system versus 2.66 for the retrained first stage and 2.86 for an ablation without noisy conditioning. The paper also compares with published DeepFilterNet2 (3.28) and UNIVERSE++ (3.44) and reports a favorable overall ranking.
Significance. If the central result holds, the paper demonstrates that a compact GAN second stage can improve a predictive enhancer's perceived quality without requiring multiple diffusion inference steps, which is useful for real-time and embedded speech enhancement. The evaluation uses an external challenge test set with multiple objective metrics, and the ablation isolates the contribution of noisy conditioning. Strengths include the low parameter count, the explicit algorithmic-latency statement, and the use of a public challenge protocol. However, the main empirical claim is weakened by the unexplained gap between the retrained first stage and the published DeepFilterNet2 checkpoint, and by missing statistical and reproducibility details.
major comments (4)
- [Table 1, Section 4] The baseline used to demonstrate the central improvement, 'First Stage only (retrained [11])', obtains NISQA-MOS 2.66, which is 0.62 below the publicly released DeepFilterNet2 checkpoint (3.28) evaluated on the same test set. Since the first stage is explicitly adopted from DeepFilterNet2, the paper should explain why retraining on the 2025 URGENT training set yields such a large drop; the training description in Section 3.2 reports only '45 epochs' and omits the loss weights and optimizer details. Because the second stage is trained on top of the retrained first stage, the reported gain (2.66 to 3.12) may be partly compensating for an underfit first stage. The published DeepFilterNet2 checkpoint is available, so the authors can directly test this by training the identical second stage on the published first stage and reporting the resulting NISQA-MOS; alternatively, they should show that the retrained first stage is not anomalously weak.
- [Sections 3.1 and 3.2, Eq. (3)] The second-stage generator is described only as 'adopted from OnlineSpatialNet' with two input channels, four blocks, and hidden size 16. The paper does not state whether the generator outputs a complex mask, a complex spectrogram, or a magnitude estimate; it also omits the value of beta in Eq. (3), the learning rates for generator and discriminator, the gradient-clipping threshold, and the exact weight-normalization and update schedules beyond 'every second iteration'. Without these details, the experiments are not reproducible and the contribution of the specific generator design to the NISQA gain cannot be isolated. Please provide an architecture summary (layer types, kernel sizes, skip connections, Mamba configuration) and a complete hyperparameter table.
- [Table 1 and Section 4] All metrics in Table 1 are single-point estimates with no confidence intervals or significance tests. The central claim relies on a NISQA-MOS increase of 0.46 over the first stage and an ablation difference of 0.26 for noisy concatenation, but NISQA is a non-intrusive predictor with nontrivial prediction variance, and several objective metrics in the same table move in the opposite direction (e.g., PESQ 2.07 to 2.03, word accuracy 75.51 to 75.05). At minimum, the authors should report bootstrap confidence intervals or per-utterance distributions for the principal metrics (especially NISQA-MOS, PESQ, and word accuracy) to establish that the reported improvements and non-degradations are not within the noise.
- [Section 3.2 and title] The title and abstract describe the system as real-time, but the paper reports only the 40 ms algorithmic latency from the two-frame lookahead; no real-time factor (RTF) or measured inference latency on any hardware is provided. For a streaming/embedded system, the RTF relative to real time on the target platform is a standard requirement and is load-bearing for the 'real-time' claim. Please report the RTF or wall-clock latency for at least one representative device (e.g., a notebook CPU or an embedded ARM core).
minor comments (5)
- [Section 2.4] The text states that each MelGAN discriminator 'focuses on a different frequency range', but the original MelGAN multi-scale discriminators operate on different time scales (via sub-sampling of the waveform), not frequency ranges. Please correct the description.
- [Section 3.2] The sentence 'We use the same learning rate and weight decay schedule as in [11]' is ambiguous: it appears after the description of second-stage training, but it is unclear whether it applies to the first-stage training (which is described only by '45 epochs') or only to the GAN training. Please state explicitly which learning-rate and weight-decay schedules are used for each training step.
- [Table 1] The row label 'First Stage only (retrained [11])' is confusing because the published DeepFilterNet2 row is also present. Rename it to something like 'First stage only (retrained by authors)' to make the distinction explicit.
- [Section 4] The overall ranking in Eq. (4) is computed over only five systems, so the difference between ranking scores (e.g., 2.25 vs 2.31) is coarse; the text should note that the ranking interpretation is limited by the small number of compared systems.
- [Section 4, Figure 3] The green-boxed region in Figure 3 is described only by color; if the manuscript is printed in grayscale, the comparison is hard to follow. Add a text label or a distinct line style to mark the recovered segment.
Circularity Check
No significant circularity: the central NISQA-MOS comparison is an external, empirical evaluation and is not constructed from fitted parameters or self-citations.
full rationale
The paper's central claim is an empirical performance improvement of the proposed two-stage system over its first stage, measured on the 2024 URGENT non-blind test set with external objective metrics such as NISQA-MOS, PESQ, ESTOI, SDR, and LSD. The second-stage generator is trained with the objective in Equation 3, which combines an adversarial hinge loss with a time-domain L1 term; no parameter in this objective or in the architecture is defined in terms of the reported evaluation metrics, and no fitted quantity is later relabeled as a prediction. The ablation study compares system variants (with and without noisy concatenation) on the same external test set, which is a standard empirical comparison rather than a circular derivation. The paper contains no load-bearing self-citations: references [11], [12], and [13] are external prior works whose architectures and pretrained checkpoints are reused, but the central result does not reduce to an assertion of those works. The only circularity-adjacent concern is that the retrained first-stage baseline (2.66 NISQA-MOS) is considerably weaker than the published DeepFilterNet2 checkpoint (3.28), which could weaken the force of the improvement claim; however, this is a benchmarking or experimental-design issue, not a circular one, because the measured gain is still an external empirical outcome rather than an identity. Overall, no specific reduction by construction, no fitted input renamed as a prediction, and no self-citation chain supporting the main result were found.
Assumptions & free parameters
free parameters (6)
- beta (beta in Eq. 3) =
not reported
- generator hidden size =
16
- generator number of blocks =
4
- GAN training epochs =
200
- discriminator update interval =
every 2 iterations
- lookahead frames =
2 frames (40 ms total latency)
assumptions (4)
- domain assumption The two-channel concatenation of noisy and intermediate enhanced spectrograms provides sufficient conditioning to learn P(x|y,z).
- domain assumption Training the first stage frozen and training only the second stage on the same dataset is a stable and effective realization of stochastic regeneration.
- domain assumption GAN training with three MelGAN discriminators on time-domain signals improves perceived quality without introducing damaging artifacts.
- domain assumption The objective metrics used (NISQA-MOS, PESQ, ESTOI, SDR, LSD, phoneme similarity, word accuracy) are valid proxies for speech quality and intelligibility.
Cite this review
Pith. "Pith review of DeepFilterGAN: A Full-band Real-time Speech Enhancement System with GAN-based Stochastic Regeneration." pith.science (2026). https://pith.science/paper/33KZTKGK
@misc{pith2026250523515,
author = {Pith},
title = {Pith review of: DeepFilterGAN: A Full-band Real-time Speech Enhancement System with GAN-based Stochastic Regeneration},
year = {2026},
howpublished = {\url{https://pith.science/paper/33KZTKGK}},
note = {Machine review of arXiv:2505.23515}
}
read the original abstract
In this work, we propose a full-band real-time speech enhancement system with GAN-based stochastic regeneration. Predictive models focus on estimating the mean of the target distribution, whereas generative models aim to learn the full distribution. This behavior of predictive models may lead to over-suppression, i.e. the removal of speech content. In the literature, it was shown that combining a predictive model with a generative one within the stochastic regeneration framework can reduce the distortion in the output. We use this framework to obtain a real-time speech enhancement system. With 3.58M parameters and a low latency, our system is designed for real-time streaming with a lightweight architecture. Experiments show that our system improves over the first stage in terms of NISQA-MOS metric. Finally, through an ablation study, we show the importance of noisy conditioning in our system. We participated in 2025 Urgent Challenge with our model and later made further improvements.
Figures
Forward citations
Cited by 1 Pith paper
-
Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement
DisCoGAN uses latent features from a frozen discriminative speech-enhancement encoder as conditioning for a GAN generator, improving low-SNR enhancement over existing GAN pipelines.
Reference graph
Works this paper leans on
-
[11]
DeepFilterNet2: Towards real-time speech enhancement on em- bedded devices for full-band audio,
H. Schr ¨oter, A. N. Escalante-B., T. Rosenkranz, and A. Maier, “DeepFilterNet2: Towards real-time speech enhancement on em- bedded devices for full-band audio,” in 17th International Work- shop on Acoustic Signal Enhancement (IWAENC 2022), 2022
work page 2022
-
[1]
Introduction In the deep learning based speech enhancement (SE) litera- ture, there are many approaches including RNN-based meth- ods [1], transformer-based methods [2], Generative Adversarial Network (GAN)-based methods [3] and diffusion-based models [4, 5]. Many works directly focus on denoising or dereverbera- tion tasks [4] while there is also an atte...
-
[2]
Background and Related Works A conventional single channel speech enhancement system tries to recover the clean speech signal x(t) by observing the noisy speech signal y(t) =h(t) ∗ x(t) +n(t) where h(t) is the im- pulse response of the channel andn(t) is an additive noise. This relation can also be represented in short-time Fourier Transform (STFT) domain...
arXiv 2025
-
[3]
Proposed Method We propose a full-band real-time stochastic regeneration sys- tem combining a predictive model with a generative adversarial network (GAN). Our system aims to enhance the performance of the first-stage model, DeepFilterNet2 [11], by incorporating a GAN-based second stage. The second stage is designed to recover over-suppressed speech segme...
-
[4]
Mel-spectrogram L1 loss to ensure consistency with the clean speech distribution. We then concatenate the noisy speech spectrogramY(k, f ) and the intermediate enhanced spectrogram Z(k, f ) obtained from the first stage to feed them into the second stage of our system. During second-stage training, the generator and dis- criminator compete using a hinge l...
-
[5]
It provides noise information that led to the particular inter- mediate enhanced speech, which inherently contains a rela- tionship between the noise information and the behavior of the first stage model
-
[6]
It provides the speech content that is possibly removed dur- ing the first stage process, which can be useful during the generation. As a result, the generator learns the clean speech distribu- tion conditioned on the noisy speech and the intermediate en- hanced speech, i.e. P(x|y, z). Our overall system that processes 48kHz speech samples in STFT domain ...
work page 2024
-
[7]
Spectral loss and multi-resolution spectrogram loss to pre- serve the frequency structure,
Show all 40 references
-
[8]
Local SNR loss to maintain speech intelligibility,
-
[9]
In addition to the output speech quality, another aspect of speech enhancement models is the deployment requirements
uses a GAN in a similar setting. In addition to the output speech quality, another aspect of speech enhancement models is the deployment requirements. For some speech enhancement applications such as mobile telecommunication, the SE system must be able to process streaming dat...
-
[10]
SI-SDR loss to enhance signal quality,
-
[12]
Experiments and Results We trained our proposed system with the training dataset of the 2025 Urgent Challenge [16]. This dataset is obtained from a collection of different individual datasets and contains speech samples with various sampling rates, namely 8k, 16k, 22.05k, 24k,...
2025
-
[13]
The discriminator learns to distinguish real clean speech from enhanced speech, while the generator refines its output to resemble clean speech more closely
while first stage model is frozen. The discriminator learns to distinguish real clean speech from enhanced speech, while the generator refines its output to resemble clean speech more closely. The optimization objectives are defined as follows: min Dl Ex [min(0, 1 − Dl(x))] +E...
-
[14]
The generative capabilities of the second stage and conditioning on the noisy speech along with the output of the first stage recover the over-suppressed speech content
Conclusion In this work, we propose a full-band real-time stochastic regen- eration system combining a predictive model with a GAN called DeepFilterGAN. The generative capabilities of the second stage and conditioning on the noisy speech along with the output of the first stag...
-
[15]
High fidelity speech enhancement with band-split rnn,
J. Yu, Y . Luo, H. Chen, R. Gu, and C. Weng, “High fidelity speech enhancement with band-split rnn,” 2023. [Online]. Available: https://arxiv.org/abs/2212.00406
2023 arXiv
-
[16]
D2former: A fully complex dual-path dual- decoder conformer network using joint complex masking and complex spectral mapping for monaural speech enhancement,
S. Zhao and B. Ma, “D2former: A fully complex dual-path dual- decoder conformer network using joint complex masking and complex spectral mapping for monaural speech enhancement,”
-
[17]
CMGAN: Conformer-based Metric GAN for Speech Enhancement,
R. Cao, S. Abdulatif, and B. Yang, “CMGAN: Conformer-based Metric GAN for Speech Enhancement,” in Proc. Interspeech 2022, 2022, pp. 936–940
2022
-
[18]
Speech enhancement and dereverberation with diffusion-based generative models,
J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2351–2364, 2023
2023
-
[19]
Storm: A diffusion-based stochastic regeneration model for speech en- hancement and dereverberation,
J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, “Storm: A diffusion-based stochastic regeneration model for speech en- hancement and dereverberation,”IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , vol. 31, pp. 2724–2737, 2023
2023
-
[20]
Universal score-based speech enhancement with high content preservation,
R. Scheibler, Y . Fujita, Y . Shirahata, and T. Komatsu, “Universal score-based speech enhancement with high content preservation,”
-
[21]
Performance measure- ment in blind audio source separation,
E. Vincent, R. Gribonval, and C. Fevotte, “Performance measure- ment in blind audio source separation,”IEEE Transactions on Au- dio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462– 1469, 2006
2006
-
[22]
Schr ¨odinger bridge for generative speech enhancement,
A. Juki ´c, R. Korostik, J. Balam, and B. Ginsburg, “Schr ¨odinger bridge for generative speech enhancement,” in Interspeech 2024, 2024, pp. 1175–1179
2024
-
[23]
Pfgm++ combined with stochastic regener- ation for speech enhancement,
X. Cao and S. Zhao, “Pfgm++ combined with stochastic regener- ation for speech enhancement,” in 2024 9th International Confer- ence on Signal and Image Processing (ICSIP), 2024, pp. 267–271
2024
-
[24]
Spectral oversubtraction? an approach for speech enhancement after robot ego speech filtering in semi-real-time,
Y . Li, K. V . Hindriks, and F. A. Kunneman, “Spectral oversubtraction? an approach for speech enhancement after robot ego speech filtering in semi-real-time,” 2024. [Online]. Available: https://arxiv.org/abs/2409.06274
2024 arXiv
-
[25]
Combined generative and predictive modeling for speech super-resolution,
H. Wang, E. W. Healy, and D. Wang, “Combined generative and predictive modeling for speech super-resolution,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14269
2024 arXiv
-
[27]
Multichannel long-term streaming neural speech enhancement for static and moving speakers,
C. Quan and X. Li, “Multichannel long-term streaming neural speech enhancement for static and moving speakers,” 2024. [Online]. Available: https://arxiv.org/abs/2403.07675
2024 arXiv
-
[28]
Melgan: Generative adversarial networks for conditional waveform synthesis,
K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y . Bengio, and A. Courville, “Melgan: Generative adversarial networks for conditional waveform synthesis,” 2019. [Online]. Available: https://arxiv.org/ abs/1910.06711
2019 arXiv
-
[29]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2024. [Online]. Available: https: //arxiv.org/abs/2312.00752
2024 arXiv
-
[30]
Ur- gent challenge: Universality, robustness, and generalizability for speech enhancement,
W. Zhang, R. Scheibler, K. Saijo, S. Cornell, C. Li, Z. Ni, J. Pirkl- bauer, M. Sach, S. Watanabe, T. Fingscheidt, and Y . Qian, “Ur- gent challenge: Universality, robustness, and generalizability for speech enhancement,” inInterspeech 2024, 2024, pp. 4868–4872
2024
-
[31]
Interspeech 2025 URGENT speech enhancement challenge,
K. Saijo, W. Zhang, S. Cornell, R. Scheibler, C. Li, Z. Ni, A. Ku- mar, M. Sach, Y . Fu, W. Wang, T. Fingscheidt, and S. Watanabe, “Interspeech 2025 URGENT speech enhancement challenge,”Ac- cepted by Interspeech, 2025
2025
-
[32]
Investigating rnn-based speech enhancement methods for noise- robust text-to-speech,
C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Investigating rnn-based speech enhancement methods for noise- robust text-to-speech,” in9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), 2016, pp. 146–152
2016
-
[33]
Icassp 2022 deep noise suppression challenge,
H. Dubey, V . Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper, and R. Aichner, “Icassp 2022 deep noise suppression challenge,”
2022
-
[35]
Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,
G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,” in Interspeech 2021, 2021, pp. 2127–2131
2021
-
[36]
Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,
A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE In- ternational Conference on Acoustics, Speech, and Signal Process- ing. Proceedings (...
2001
-
[38]
Distance measures for speech process- ing,
A. Gray and J. Markel, “Distance measures for speech process- ing,” IEEE Transactions on Acoustics, Speech, and Signal Pro- cessing, vol. 24, no. 5, pp. 380–391, 1976
1976
-
[39]
An algorithm for predicting the in- telligibility of speech masked by modulated noise maskers,
J. Jensen and C. H. Taal, “An algorithm for predicting the in- telligibility of speech masked by modulated noise maskers,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 24, no. 11, pp. 2009–2022, 2016
2009
-
[40]
Evaluation metrics for generative speech en- hancement methods: Issues and perspectives,
J. Pirklbauer, M. Sach, K. Fluyt, W. Tirry, W. Wardah, S. Moeller, and T. Fingscheidt, “Evaluation metrics for generative speech en- hancement methods: Issues and perspectives,” inSpeech Commu- nication; 15th ITG Conference, 2023, pp. 265–269
2023
-
[2022]
Available: https://arxiv.org/abs/2202.13288
[Online]. Available: https://arxiv.org/abs/2202.13288
-
[2023]
Available: https://arxiv.org/abs/2302.11832
[Online]. Available: https://arxiv.org/abs/2302.11832
-
[2024]
Available: https://arxiv.org/abs/2406.12194
[Online]. Available: https://arxiv.org/abs/2406.12194
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.