REVIEW 4 major objections 6 minor 36 references
Model as Loss: A Self-Consistent Training Paradigm
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that adding an L1 loss between the model's own encoder embeddings of clean and enhanced speech improves quality and generalization beyond handcrafted and pretrained WavLM feature losses.
desk verdict Model as Loss is a genuinely new training trick with plausible gains, but the paper under-delivers on isolating why it works. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the MAL-encoder: the bottleneck feature extractor of the same speech enhancement model, used as a fixed or slowly updated loss network after the model has been trained with spectral losses. Its output feeds the identity $L_{\text{mal}} = \| \text{Encoder}(y_{\text{clean}}) - \text{Encoder}(y_{\text{enhanced}}) \|_1$, guiding the decoder to produce signals whose embeddings match the clean reference. The paper's three variants control how the MAL-encoder is updated: frozen with only the decoder trained, frozen while the whole encoder-decoder trains, and dynamic with the encoder refreshed every epoch. This mechanism turns the model's own learned representation into a perceptual and task-specific distance, and the paper argues it also enforces self-consistency under repeated enhancement.
What would settle it
Train the same DeepFilterNet2 baseline with $L_{\text{mal}}$ but replace the MAL-encoder with an encoder that has only seen clean speech and therefore cannot represent noise; if the resulting model does not improve over the spectral-loss baseline on NISQA or PESQ, then the loss's benefit depends on the encoder's noise sensitivity rather than on a universal feature-space distance.
Extended reading notes
Core claim
The central discovery is that minimizing $L_{\text{mal}} = \| \text{Encoder}(y_{\text{clean}}) - \text{Encoder}(y_{\text{enhanced}}) \|_1$ as an auxiliary objective improves speech enhancement over conventional spectral losses and over deep feature losses computed from a pretrained WavLM. After training a DeepFilterNet2 baseline with its standard spectral loss, the authors add this encoder-embedding loss in equal proportion and fine-tune under three variants: a frozen encoder that trains only the decoder, a frozen encoder that trains the full model, and a dynamic encoder that is refreshed each epoch. All three variants beat the WavLM-based losses on NISQA and ScoreQ metrics, with the dynamic variant highest on NISQA and the frozen full-model variant best on intrusive metrics such as PESQ and ESTOI. The method also preserves speech better under iterative enhancement, indicating a self-consistency property in the encoder's feature space.
Load-bearing premise
The method assumes that L1 distance in the encoder's learned bottleneck space tracks perceptual quality, so that pulling the enhanced signal's embedding toward the clean signal's embedding actually removes audible noise rather than only matching feature statistics.
Editorial extensions
If this is right
- Speech enhancement models can be improved without large pretrained feature extractors: a 2.31-million-parameter model's own encoder outperforms a 95-million-parameter WavLM-based loss.
- MAL improves out-of-domain generalization; the dynamic and frozen variants show larger gains on unseen Urgent Challenge sets than on in-domain DNS sets.
- Models trained with MAL remain stable under iterative enhancement, preserving harmonics for many more passes than spectral-loss or WavLM baselines.
- The paradigm transfers to other encoder-decoder restoration tasks such as echo cancellation, image denoising, or medical image analysis, as argued in the conclusion.
Reading between the lines
- If the encoder's embedding distance is a good perceptual surrogate only after the model has already learned to suppress noise, then MAL is best understood as a second-stage refinement regularizer rather than a standalone training objective.
- The dynamic variant resembles self-training or consistency regularization, so its success suggests that updating the loss network each epoch keeps the feature space aligned with the current model; a testable extension is to vary the update frequency to find the trade-off between stability and alignment.
- A natural stress test is to apply MAL to an encoder-decoder model in image denoising, where the encoder has no pretrained speech priors; if gains appear there too, the principle is domain-general as the paper hopes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Model as Loss (MAL), a training paradigm for encoder-decoder speech enhancement in which the model's own encoder embeddings are used as an additional loss term, L_mal = ||Encoder(y_clean) - Encoder(y_enhanced)||_1, after an initial training stage with conventional spectral losses. Three variants are introduced: a frozen MAL-encoder with decoder-only fine-tuning, a frozen MAL-encoder with full-model fine-tuning, and a per-epoch updated ('dynamic') MAL-encoder. Experiments on DeepFilterNet2 compare these variants against the baseline, the baseline fine-tuned for extra epochs, a WavLM deep-feature loss, and combinations thereof, using NISQA, ScoreQ, SIGMOS, PESQ, ESTOI, LSD, and MCD on in-domain and out-of-domain sets. The paper reports consistent improvements for MAL variants and analyzes iterative enhancement stability.
Significance. If the reported results are robust, MAL is an attractive and cost-effective alternative to pre-trained deep feature losses, since it removes the need for an external feature extractor and can in principle transfer to domains without pre-trained models. The paper includes a credible in-domain/out-of-domain evaluation, an ablation on starting encoder quality, and an iterative-enhancement experiment that directly addresses self-consistency. The significance is currently limited by the absence of uncertainty quantification, by the confounded comparison between MAL and WavLM losses, and by missing controls that would isolate the mechanism of the gain.
major comments (4)
- [Section 3] The comparison between L_mal and L_wavlm is confounded by loss scale: the paper states that L_mal and L_wavlm are added 'in equal proportion' to the DeepFilterNet2 loss, but no per-loss weight tuning, gradient normalization, or gradient-norm statistics are reported. Since the two auxiliary losses operate in different feature spaces with different dimensionalities and magnitudes, the observed superiority of L_mal could result from a more favorable loss scale rather than from the encoder's task-specific feature space. Please report gradient norms and include a control in which the WavLM loss is scaled (e.g., via gradient-norm matching) to the same magnitude as L_mal.
- [Section 2, Eq. (3)] The central claim that the encoder's learned feature space is responsible for the improvement is not isolated: there is no control where Encoder in Eq. (3) is replaced by a fixed random feature map of identical output dimension and loss scale, or by a fixed non-task-specific feature extractor. Without such a control, the gain could be attributed to the auxiliary loss acting as an additional regularizer rather than to the perceptual relevance of the encoder. Please add a random-feature or fixed-encoder baseline to support the mechanism.
- [Section 2, Ours_mal-dynamic] The L_mal-dynamic variant updates the MAL-encoder every epoch while training the full model, which creates a moving target for the loss. Because the encoder can change so that embeddings of clean and enhanced speech become close regardless of audio quality, the dynamic variant is susceptible to representation collapse, yet the paper does not analyze encoder drift, feature-space geometry, or training stability beyond one sentence about per-batch updates in Section 4.1. Please quantify the drift of the MAL-encoder over fine-tuning and compare dynamic versus frozen variants at matched total compute and epochs, or provide an argument for why collapse cannot occur.
- [Section 3.2, Tables 1 and 2] The paper states that ANOVA analysis was performed and only statistically significant metrics are reported, but Tables 1 and 2 contain no significance markers, confidence intervals, effect sizes, or information about the number of training runs. Given the small absolute gains (e.g., NISQA Overall 3.72 for Ours_mal-dynamic versus 3.56 for Baseline_wavlm in-domain), the empirical claim requires reproducibility measures such as bootstrap intervals or multiple random initializations. Please add these or clearly state the statistical analysis used.
minor comments (6)
- [Section 3.2] The text contains a typo: 'ANOV A' should be 'ANOVA'.
- [Section 3] The window name is misspelled as 'V orbis'; it should be 'Vorbis'.
- [Figure 1] The spectrogram figure is difficult to read at the printed scale; consider zooming into a specific frequency band or adding annotations to highlight the preserved harmonics.
- [Section 4.1] The claim that 'the better the encoder, the more effective L_mal becomes' is supported by only two encoder configurations; two data points do not establish a monotonic trend.
- [Section 5] The conclusion states that MAL is 'domain-agnostic', but all experiments are in speech enhancement; please temper this claim unless additional domains are tested.
- [References] Reference [6] has inconsistent formatting: it mixes a Proceedings of Interspeech citation with a 2018 arXiv-style year and includes page numbers in an unusual way.
Circularity Check
No significant circularity: MAL is a training objective evaluated against external benchmarks, not a result derived from its own definition.
full rationale
The paper's central claim is empirical: adding Lmal = ||Encoder(clean) - Encoder(enhanced)||1 to the DeepFilterNet2 loss improves perceptual quality and generalization compared with handcrafted losses, WavLM deep feature losses, and additional baseline epochs. The results in Tables 1 and 2 are measured against external evaluation metrics (NISQA, SIGMOS, ScoreQ, PESQ, ESTOI) and external test sets, not derived from the loss definition itself. The Section 2 discussion of an 'ideal loss' satisfying LF = ||F(yclean) - F(yenhanced)||1 for every F is a definition/tautology, but the paper does not use it to prove performance; it motivates the encoder-as-loss objective. The claim that models trained with MAL better preserve speech under iterative enhancement is an observed behavior (Figures 1 and 3), not a forced consequence of the equations. There are no load-bearing self-citations: the cited related work (WavLM, DeepFilterNet2, Babaev et al.) is external, and no uniqueness theorem or prior result by the same authors is invoked to exclude alternatives. The absence of a random-feature control or loss-weight calibration is an experimental confound and a threat to the mechanistic interpretation, but it is not circularity: the reported improvements are not equivalent to the inputs by construction. The assumption that the encoder's feature space is perceptually relevant is an empirical hypothesis, not a self-referential derivation. Overall, the claimed results are self-contained empirical findings, and no step reduces to its own inputs.
Assumptions & free parameters
free parameters (3)
- Loss weighting coefficient =
1.0 (equal proportion)
- Finetuning epochs =
10
- MAL warmup period N =
10 epochs
assumptions (3)
- domain assumption L1 distance in the encoder bottleneck feature space is a valid perceptual quality measure.
- domain assumption The encoder trained with conventional spectral losses already extracts features relevant to speech enhancement.
- domain assumption Clean and enhanced utterances can be mapped to the same point in the encoder feature space without introducing artifacts.
Cite this review
Pith. "Pith review of Model as Loss: A Self-Consistent Training Paradigm." pith.science (2026). https://pith.science/paper/3MLOXMKF
@misc{pith2026250521156,
author = {Pith},
title = {Pith review of: Model as Loss: A Self-Consistent Training Paradigm},
year = {2026},
howpublished = {\url{https://pith.science/paper/3MLOXMKF}},
note = {Machine review of arXiv:2505.21156}
}
read the original abstract
Conventional methods for speech enhancement rely on handcrafted loss functions (e.g., time or frequency domain losses) or deep feature losses (e.g., using WavLM or wav2vec), which often fail to capture subtle signal properties essential for optimal performance. To address this, we propose Model as Loss, a novel training paradigm that utilizes the encoder from the same model as a loss function to guide the training. The Model as Loss paradigm leverages the encoder's task-specific feature space, optimizing the decoder to produce output consistent with perceptual and task-relevant characteristics of the clean signal. By using the encoder's learned features as a loss function, this framework enforces self-consistency between the clean reference speech and the enhanced model output. Our approach outperforms pre-trained deep feature losses on standard speech enhancement benchmarks, offering better perceptual quality and robust generalization to both in-domain and out-of-domain datasets.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Speech enhancement has long been a challenging problem, with applications in telecommunication, hearing aids, and robust au- tomatic speech recognition (ASR) [1, 2, 3, 4, 5]. A critical component in training enhancement models is the choice of loss function, which directly influences the quality and generaliza- tion of enhanced output [6, 7]....
-
[2]
Model as Loss: A Self-Consistent Training Paradigm
Methodology In the realm of speech enhancement, models typically comprise an encoder and one or more decoders [5]. For simplicity, we will refer to this setup as a single encoder-decoder system mov- ing forward. The encoder’s job is to extract relevant features from the noisy signal, which are then used by the decoder to synthesize the enhanced signal. Th...
work page Pith review arXiv 2025
-
[3]
It has an encoder that extracts relevant fea- tures and passes them into a first-stage decoder
Experimental setup We base all our experiments on DeepFilterNet2 proposed by Schr¨oteret al.[5]. It has an encoder that extracts relevant fea- tures and passes them into a first-stage decoder. The output of this decoder is passed into the deep filtering decoder, which predicts the deep filtering coefficients for each time frame. We train DeepFilterNet2 as...
work page 2024
-
[4]
Results Tables 1 and 2 present all metrics, clearly demonstrating that the proposed models withL mal losses outperform the others. Oursmal−dynamic achieves the best performance across all NISQA metrics, while Ours mal−frozen leads in all intrusive metrics. The Oursmal−frozen-fe model outperforms others in SIGMOS Signal and Overall metrics, while performin...
work page 2025
-
[5]
Conclusion In this paper, we proposeModel as Loss (MAL), a novel train- ing paradigm that leverages the encoder of an encoder-decoder model as a loss function to guide optimization. By align- ing the loss with the model’s task-specific feature space,MAL overcomes the limitations of traditional handcrafted and pre- trained deep feature losses. This approac...
-
[6]
Acknowledgements The authors thank Paul Kendrik, Tijana Stojkovic, and Andy Pearce for their valuable feedback and insights. We also thank Sai Dhawal Phaye for discussions during the early stages of MAL, and Kanav Sabharwal for his feedback on the writing
-
[7]
J. Benesty, S. Makino, and J. Chen,Speech enhancement. Springer Science & Business Media, 2006
work page 2006
-
[8]
C. K. Reddy, V . Gopal, R. Cutler, E. Beyrami, R. Cheng, H. Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun et al., “The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,” arXiv preprint arXiv:2005.13981, 2020
arXiv 2020
Show all 36 references
-
[9]
FINALLY: fast and universal speech enhancement with studio- like quality,
N. Babaev, K. Tamogashev, A. Saginbaev, I. Shchekotov, H. Bae, H. Sung, W. Lee, H.-Y . Cho, and P. Andreev, “FINALLY: fast and universal speech enhancement with studio- like quality,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online...
2024
-
[10]
Icassp 2023 deep noise suppression challenge,
H. Dubey, A. Aazami, V . Gopal, B. Naderi, S. Braun, R. Cutler, A. Ju, M. Zohourian, M. Tang, H. Gamper, M. Golestaneh, and R. Aichner, “Icassp 2023 deep noise suppression challenge,”
2023
-
[11]
Wavlm: Large-scale self-supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”CoRR, vol. abs/2110.13900, 2021...
-
[12]
DeepFilterNet2: Towards real-time speech enhancement on em- bedded devices for full-band audio,
H. Schr ¨oter, A. N. Escalante-B., T. Rosenkranz, and A. Maier, “DeepFilterNet2: Towards real-time speech enhancement on em- bedded devices for full-band audio,” in17th International Work- shop on Acoustic Signal Enhancement (IWAENC 2022), 2022. [Online]. Available: https://gi...
2022
-
[13]
Speech denoising with deep feature losses,
F. G. Germain, Q. Chen, and V . Koltun, “Speech denoising with deep feature losses,”Proc. Interspeech 2019, 2723-2727, 2018
2019
-
[14]
A consolidated view of loss functions for supervised deep learning-based speech enhancement,
S. Braun and I. Tashev, “A consolidated view of loss functions for supervised deep learning-based speech enhancement,” in2021 44th International Conference on Telecommunications and Signal Processing (TSP). IEEE, 2021, pp. 72–76
2021
-
[15]
auraloss: Audio focused loss functions in PyTorch,
C. J. Steinmetz and J. D. Reiss, “auraloss: Audio focused loss functions in PyTorch,” inDigital Music Research Network One- day Workshop (DMRN+15), 2020
2020
-
[16]
A deep learning loss function based on the perceptual evaluation of the speech quality,
J. Mart ´ın-Do˜nas, A. Gomez, J. Gonzalez Lopez, and A. Peinado, “A deep learning loss function based on the perceptual evaluation of the speech quality,”IEEE Signal Processing Letters, vol. PP, pp. 1–1, 09 2018
2018
-
[17]
On loss func- tions for supervised monaural time-domain speech enhancement,
M. Kolbæk, Z.-H. Tan, S. H. Jensen, and J. Jensen, “On loss func- tions for supervised monaural time-domain speech enhancement,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 28, pp. 825–838, 2020
2020
-
[18]
Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,
G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,” Aug. 2021. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2021-299
2021 doi
-
[19]
wav2vec: Unsupervised pre-training for speech recognition,
S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,”arXiv preprint arXiv:1904.05862, 2019
1904 arXiv
-
[20]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,”CoRR, vol. abs/2006.11477, 2020. [Online]. Available: https://arxiv.org/abs/2006.11477
2006 arXiv
-
[21]
Speechlmscore: Evaluating speech generation using speech language model,
S. Maiti, Y . Peng, T. Saeki, and S. Watanabe, “Speechlmscore: Evaluating speech generation using speech language model,”
-
[22]
Enhancing lora reception with generative models: Channel-aware denoising of loraphy signals,
K. Sabharwal, S. Ramesh, J. Wang, D. M. Divakaran, and M. C. Chan, “Enhancing lora reception with generative models: Channel-aware denoising of loraphy signals,” inProceedings of the 22nd ACM Conference on Embedded Networked Sensor Sys- tems, 2024, pp. 507–520
2024
-
[23]
Icassp 2022 deep noise suppression challenge,
H. Dubey, V . Gopal, R. Cutler, A. Aazami, S. Matusevych, S. Braun, S. E. Eskimez, M. Thakker, T. Yoshioka, H. Gamper, and R. Aichner, “Icassp 2022 deep noise suppression challenge,”
2022
-
[24]
Available: https://arxiv.org/abs/2202.13288
[Online]. Available: https://arxiv.org/abs/2202.13288
-
[25]
Urgent challenge: Universality, robustness, and generalizability for speech enhance- ment,
W. Zhang, R. Scheibler, K. Saijo, S. Cornell, C. Li, Z. Ni, A. Ku- mar, J. Pirklbauer, M. Sach, S. Watanabeet al., “Urgent challenge: Universality, robustness, and generalizability for speech enhance- ment,”arXiv preprint arXiv:2406.04660, 2024
2024 arXiv
-
[26]
Icassp 2024 speech signal improvement challenge,
N. C. Ristea, A. Saabas, R. Cutler, B. Naderi, S. Braun, and S. Branets, “Icassp 2024 speech signal improvement challenge,”
2024
-
[29]
Scoreq: Speech quality assessment with contrastive regression,
A. Ragano, J. Skoglund, and A. Hines, “Scoreq: Speech quality assessment with contrastive regression,”arXiv preprint arXiv:2410.06675, 2024
2024 arXiv
-
[30]
Do- main adaptation and autoencoder-based unsupervised speech enhancement,
Y . Li, Y . Sun, K. Horoshenkov, and S. M. Naqvi, “Do- main adaptation and autoencoder-based unsupervised speech enhancement,”IEEE Transactions on Artificial Intelligence, vol. 3, no. 1, p. 43–52, Feb. 2022. [Online]. Available: http://dx.doi.org/10.1109/TAI.2021.3119927
2022
-
[31]
Medical image denoising using convolutional de- noising autoencoders,
L. Gondara, “Medical image denoising using convolutional de- noising autoencoders,” in2016 IEEE 16th international confer- ence on data mining workshops (ICDMW). IEEE, 2016, pp. 241–246
2016
-
[33]
Neural cascade architecture for multi- channel acoustic echo suppression,
H. Zhang and D. Wang, “Neural cascade architecture for multi- channel acoustic echo suppression,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2326–2336, 2022
2022
-
[34]
Task splitting for dnn-based acous- tic echo and noise removal,
S. Braun and M. L. Valero, “Task splitting for dnn-based acous- tic echo and noise removal,” in2022 International Workshop on Acoustic Signal Enhancement (IWAENC). IEEE, 2022, pp. 1–5
2022
-
[35]
Deep learning on image denoising: An overview,
C. Tian, L. Fei, W. Zheng, Y . Xu, W. Zuo, and C.-W. Lin, “Deep learning on image denoising: An overview,”Neural Networks, vol. 131, pp. 251–275, 2020
2020
-
[36]
A review of the deep learning methods for medical images super resolution problems,
Y . Li, B. Sixou, and F. Peyrin, “A review of the deep learning methods for medical images super resolution problems,”Irbm, vol. 42, no. 2, pp. 120–133, 2021
2021
-
[2022]
Available: https://arxiv.org/abs/2212.04559
[Online]. Available: https://arxiv.org/abs/2212.04559
-
[2023]
Available: https://arxiv.org/abs/2303.11510
[Online]. Available: https://arxiv.org/abs/2303.11510
-
[2024]
Available: https://arxiv.org/abs/2401.14444
[Online]. Available: https://arxiv.org/abs/2401.14444
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.