{"id":"35d5e0ec-2482-4204-a414-79a8dcb362ad","arxiv_id":"2506.11514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Denoising audio embeddings from a generative pretrained encoder before vocoder synthesis yields efficient speech enhancement with better speaker fidelity than discriminative encoders.","lead":"This paper builds a speech enhancement system that removes noise from the embedding representation produced by a frozen pretrained audio encoder, then reconstructs speech with a pretrained vocoder. The authors report that a generative audio encoder preserves speaker identity better than discriminative encoders and that their system beats the Demucs baseline in a small listening test.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison rests on an untested assumption that Vocos can faithfully invert MSE-denoised Dasheng embeddings; the paper never verifies this reconstruction path.","rationale":"The reader identified the invertibility assumption as the weakest point, and I agree. The paper's central claim is that denoising in Dasheng space and synthesizing with Vocos outperforms Demucs and discriminative encoders. All of this depends on Vocos being a reliable inverter of the frozen encoder, and on MSE denoising keeping the embedding in the region where Vocos was trained. Neither is demonstrated. A clean-embedding reconstruction control is a cheap, decisive experiment: it establishes the upper bound of the pipeline and distinguishes reconstruction fidelity from the GAN prior. The small MOS test and outdated Demucs baseline are additional concerns, but they are secondary; even a much larger listening test would not resolve whether the method's success comes from the embedding trick or from the vocoder's ability to generate pleasant but unfaithful speech. The released code and checkpoint make the proposed control feasible. Therefore the verdict stays CONDITIONAL, unchanged by this pass.","tokens_in":7835,"tokens_out":7925,"duration_ms":79313,"concrete_test":"Run the released inference code on the DNS1 test set with a clean-embedding control: for each utterance, skip the denoiser and synthesize Vocos(z_clean) directly from the clean Dasheng embedding. Measure PESQ, STOI, and ECAPA cosine between Vocos(z_clean) and the original clean speech, and compare with the same metrics for Vocos(z_denoised). If Vocos(z_clean) itself fails to reach high fidelity, or if denoised-embedding synthesis is substantially worse than clean-embedding synthesis, then the vocoder does not faithfully invert the embedding space and the claimed advantage cannot be attributed to the proposed denoising strategy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step of the method is not the denoise encoder itself but the implicit claim that the Dasheng embedding is invertible for out-of-distribution inputs. In Section 2 (Figure 1c), Vocos is trained only on clean-speech embeddings; at inference (Figure 1a) it must synthesize from MSE-denoised noisy embeddings. MSE in a 768-dimensional space does not guarantee that z_d = DE(z_noisy) lies on the clean-embedding manifold: minimizing L2 to a single target can produce an average of plausible clean embeddings that is itself off-manifold. The paper provides no reconstruction sanity check for Vocos on clean embeddings, no analysis of the distribution of z_d relative to clean embeddings, and no ablation that would separate 'the embedding is invertible' from 'Vocos generates pleasant audio regardless of conditioning.' The internal evidence actually makes this concern concrete: in Table 1, PESQ/STOI are at or below the noisy reference for most configurations, and even the best Dasheng system loses STOI (0.90 vs 0.92) on both test sets. If Vocos is generating from off-manifold embeddings, the subjective MOS advantage over Demucs could reflect the GAN vocoder's prior rather than a faithful reconstruction of the cleaned speech, which would undermine the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a speech enhancement system that operates in the embedding domain: a frozen pre-trained audio encoder extracts embeddings from noisy speech, a compact denoise encoder maps them toward clean-speech embeddings under an MSE loss, and a pre-trained Vocos vocoder synthesizes the waveform from the denoised embeddings. The authors compare three audio encoders (WavLM, Whisper, and Dasheng) plus a log-mel spectrogram baseline, report intrusive and non-intrusive quality metrics, speaker-fidelity results, a parameter/architecture ablation of the denoise encoder, and a subjective listening test against the Demucs enhancement model. The central claim is that the generative audio encoder Dasheng yields better perceptual quality (MOS 3.87 vs 3.11 for Demucs) and better speaker fidelity than the discriminative encoders, while the tiny denoise encoder (as small as 1.2M parameters) suffices for competitive performance.","tokens_in":8031,"tokens_out":3900,"duration_ms":39269,"significance":"If the claims hold, the paper contributes a simple, parameter-efficient recipe for speech enhancement that leverages frozen generative audio encoders, and it draws a useful empirical distinction between discriminative and generative embedding spaces. The study is strengthened by several concrete elements: the audio encoder and vocoder are frozen during denoise-encoder training, ablations systematically vary denoise-encoder size and architecture, a subjective listening test is included, and the authors release code, a checkpoint, and audio examples. The main value is not a new state-of-the-art result but a well-scoped demonstration that embedding-domain enhancement with a generative encoder can preserve speaker identity and perceived quality better than embedding-domain enhancement with discriminative encoders. This significance is conditional, however, on verifying that the vocoder faithfully inverts the denoised embeddings, because the current evidence leaves open the possibility that the subjective advantage comes from the vocoder's generative prior rather than from accurate reconstruction of cleaned speech.","major_comments":[{"comment":"The load-bearing reconstruction path is not validated. Vocos is trained only on clean-speech embeddings (Figure 1c), but at inference it must synthesize from MSE-denoised embeddings of noisy speech; MSE in 768-dimensional space does not guarantee that the denoised embedding lies on the clean-embedding manifold. Please add (a) a reconstruction sanity check in which clean embeddings are fed to the trained vocoder and the output is compared to the original clean waveform with PESQ/STOI/DNSMOS, (b) a distributional analysis of denoised versus clean embeddings (e.g., nearest-neighbor distances or a simple manifold statistics report), and (c) a control ablation that feeds unprocessed noisy embeddings directly to the vocoder, so the reader can separate the contribution of the vocoder prior from the contribution of the denoise encoder. Without one of these checks, the subjective MOS advantage over Demucs could be produced by Vocos generating pleasant audio regardless of whether the denoised embedding faithfully represents the cleaned speech.","section":"Section 2, Figure 1, Section 3.2"},{"comment":"The standard intrusive metrics in Table 1 do not support the central enhancement claim. On Valentini, the Dasheng system improves PESQ over noisy (2.32 vs 1.97) but reduces STOI (0.90 vs 0.92), and both WavLM and Whisper degrade both PESQ and STOI substantially relative to noisy; on DNS1, WavLM and Whisper degrade both metrics, while Dasheng only ties STOI (0.92 vs 0.92). The text in Section 4.1 states that audioencoder-based methods show 'distinct performance improvement' in noise suppression, but that conclusion relies exclusively on non-intrusive metrics (DNSMOS, NISQAv2), and the paper does not explain why the intrusive metrics disagree so sharply. Please report per-sample statistics or significance tests for the PESQ/STOI differences, discuss the intelligibility loss as a limitation, and justify why DNSMOS/NISQAv2 should be given priority over PESQ/STOI for this system.","section":"Section 4.1, Table 1"},{"comment":"The subjective listening test is central to the paper's main claim but is underreported. The paper gives 17 listeners and 15 audio samples but no test methodology (e.g., MUSHRA, ITU-T P.808, or a paired-comparison protocol), no confidence intervals or statistical significance test for the 0.76 MOS gap versus Demucs, no description of sample selection or SNR distribution, and no inter-rater reliability measure. Please add these details and a significance analysis. In addition, the Demucs baseline should be justified as an appropriate 'state-of-the-art' speech enhancement system, and ideally the subjective comparison should include at least one recent SE model; the reference as written (Défossez et al., 'Real time speech enhancement in the waveform domain') is several years old.","section":"Section 4.3, Table 5"},{"comment":"The speaker-fidelity comparison in Table 4 is missing a critical reference: the ECAPA-TDNN cosine similarity between the clean speaker embedding and the embedding of the unprocessed noisy speech. Without this baseline, the low values for WavLM and Whisper could be caused by vocoder resynthesis artifacts or by the enhancement pipeline's signal distortion rather than by loss of speaker information in the embedding itself. Please report the speaker similarity of the noisy input and of the LMS baseline (the latter is already in the table but its relationship to the claim is not discussed), and consider also reporting a speaker-verification equal-error rate or at least error bars across the test utterances.","section":"Section 4.1, Table 4"}],"minor_comments":[{"comment":"There is a typo in the Introduction: 'robust auudioencoders' should be 'robust audioencoders'.","section":"Section 1"},{"comment":"The sentence 'This technique simulates noisy conditions and ensures the that the model is robust to different noise environments' contains a duplicated article; please simplify to 'ensures that the model is robust...'.","section":"Section 3.1"},{"comment":"The paragraph on NISQAv2 ends with the fragment 'The overall score of' before the sentence is cut off. Please complete the sentence and state explicitly whether the overall NISQAv2 score or a subscore is reported in Tables 1 and 3.","section":"Section 3.3"},{"comment":"Several cells in Table 1 have run-together numeric values (e.g., '3.493.494.133.26 4.12' in the WavLM row on Valentini). Please format all rows so that each metric column contains exactly one number.","section":"Table 1"},{"comment":"The color coding of fine-tuned (red) versus frozen (blue) parameters may not survive print or grayscale rendering; consider adding a symbol or footnote to distinguish these cases.","section":"Table 2"},{"comment":"The phrase 'an existing state-of-the-art SE model' in the abstract and Section 4.3 is only supported by a single, older baseline (Demucs/Denoiser); please temper the wording or add more recent baselines.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a compact empirical paper whose central claim is plausible but currently rests on an unverified reconstruction assumption and on non-intrusive metrics that conflict with PESQ/STOI. The requested additions (vocoder reconstruction sanity check, noisy-baseline speaker fidelity, subjective-test statistics) are feasible within the paper's scope and would substantially raise the reliability of the conclusions. I do not see evidence of circularity or fabricated results; the main issue is missing validation of the pipeline's most critical path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, well-scoped system paper. The framework—denoise embeddings from a frozen encoder, synthesize with a pretrained vocoder—is not new; citations [6]–[8] cover it. What is new is the systematic comparison between generative (Dasheng) and discriminative (WavLM, Whisper) audioencoders in that framework, and the conclusion that generative embeddings preserve speaker identity much better. The parameter-efficiency ablation is also refreshingly honest: even a 1.2M-parameter MLP denoiser gives decent non-intrusive scores, and the released code and checkpoints are a real plus.\n\nMain soft spots are in evaluation. The subjective test uses 17 listeners and 15 samples, with no variance or significance testing, and \"state-of-the-art\" is represented by Demucs (2020). The paper also reports that STOI drops below the noisy reference for most configurations, including Dasheng; the authors acknowledge this but do not explain it. The stress-test's invertibility concern is legitimate: they never show that MSE-denoised 768-dimensional embeddings lie on the clean-embedding manifold, or that the Vocos vocoder reconstructs faithfully from off-manifold inputs. That said, the speaker-fidelity results (cosine similarity 0.78–0.88 for Dasheng) are consistent with the embeddings retaining speaker information, which partly mitigates the concern. Still, a simple reconstruction sanity check—clean embeddings through Vocos vs. original audio, and a distributional plot of denoised embeddings—would directly answer it and would be cheap to add.\n\nMy take: the central comparison between encoders is internally valid, and the paper's conclusion that generative embeddings preserve speaker identity is supported. The absolute MOS advantage over Demucs is a system-level result but less clean because the vocoder's generation prior is not separated. This is a solid subfield contribution, not a breakthrough. It deserves peer review, but I would send it back for a revision that adds the reconstruction check, a stronger baseline or two, and more transparent subjective results.","headline":"A clean, parameter-efficient embedding-denoiser recipe; the generative-vs-discriminative comparison is the real finding, but absolute gains rest on a vocoder whose reconstruction behavior isn't fully verified.","tokens_in":8587,"tokens_out":3353,"would_cite":false,"duration_ms":33045,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Denoising in the embedding space of a frozen generative audioencoder gives a small-network speech enhancer that beats a waveform-domain baseline in perceived quality.","keywords":["speech enhancement","audio embeddings","pre-trained audioencoder","vocoder synthesis","speaker fidelity","embedding-space denoising","parameter efficiency"],"falsifier":"A single experiment would settle the identity-preservation claim: take a set of unseen speakers, enhance their noisy speech with the full Dasheng pipeline, and run speaker verification between each clean utterance and its enhanced version; if equal-error rates are far worse than clean-versus-clean rates, the pipeline is altering speaker identity rather than preserving it.","tokens_in":7618,"feed_emoji":"🎧","tokens_out":9752,"duration_ms":88682,"temperature":0.7,"pith_summary":"The paper proposes treating speech enhancement as embedding-space denoising rather than waveform or time-frequency masking. A frozen pre-trained generative audioencoder maps noisy speech to embeddings; a compact denoise encoder (as small as two MLP layers) moves those embeddings toward clean ones; a pre-trained vocoder synthesizes the waveform. The authors claim this generative-embedding route preserves speaker and timbre information better than discriminative audioencoders, and that it surpasses the waveform-to-waveform Demucs model in subjective listening tests, with MOS 3.87 versus 3.11. The practical payoff is that neither the audioencoder nor the vocoder needs fine-tuning, leaving a lightweight, swappable module for restoration tasks.","feed_headline":"Generative-embedding denoising beats waveform SE in listener MOS","feed_subtitle":"Denoising 768-D embeddings from a frozen audioencoder preserves speaker identity and needs only a tiny network.","key_machinery":"The load-bearing mechanism is the invertibility of the frozen generative audioencoder's embedding space: the Dasheng encoder outputs a 768-dimensional embedding from which a Vocos-style vocoder trained on clean speech can resynthesize audio, so signal enhancement reduces to regression in embedding space. The denoise encoder is trained with mean squared error between embeddings of paired noisy and clean speech, while the vocoder is trained separately on clean speech alone; a compact ViT-based network, and in the ablation even two MLP layers with 1.2M parameters, performs the regression. Keeping both the audioencoder and vocoder frozen is what gives the method its parameter efficiency and makes the middle module replaceable for other tasks.","core_discovery":"The central claim is that generative audioencoder embeddings are the right substrate for vocoder-based speech enhancement, because they carry the full speech content, including timbre and pitch, that a vocoder needs to reproduce the original speaker. With the Dasheng encoder frozen and a vocoder trained only on clean embeddings, the system reports PESQ 2.32 on the Valentini set and 2.24 on the DNS1 set, and speaker-fidelity similarity of 0.783 and 0.881, whereas WavLM and Whisper embeddings, though they suppress noise well, drop speaker similarity to about 0.41 and 0.49. In a listening study the Dasheng pipeline scores 3.87 MOS against 3.11 for Demucs, which the paper takes as evidence that denoising generative embeddings produces perceptually cleaner speech without erasing the speaker.","pith_inferences":["A useful stress test would be to pass already-clean speech through the full encode-denoise-synthesize loop; the resulting drop in MOS or speaker similarity would quantify how much distortion the vocoder itself adds.","The comparison across audioencoders trains a separate vocoder for each encoder, so a cleaner decomposition would train a vocoder on WavLM or Whisper embeddings with identical effort to see whether low speaker fidelity is intrinsic to those embeddings or to the synthesis pair.","If the embedding manifold is smooth, the ablation suggests the denoise module could shrink further; the practical floor is likely where noisy embeddings can no longer be mapped back into the vocoder's clean-embedding manifold.","The method's promise for other tasks depends on whether the generative embedding space is degradation-agnostic; a direct test is to train the same denoise encoder with reverberant or separated-source targets and measure whether the vocoder output follows."],"forward_implications":["A separate, very small network can perform speech enhancement once a generative audioencoder and vocoder already exist; the ablation shows even 1.2M-parameter MLP2 produces usable perceptual quality.","The same pre-trained vocoder works across different denoise encoders, so scaling or swapping the denoising module does not require retraining synthesis.","The vocoder is trained from clean speech only, so the demand for paired noisy/clean data is confined to the lightweight denoise encoder.","Because generative embeddings are claimed to retain timbre, the same encoder-vocoder pair should transfer to dereverberation, source separation, and bandwidth expansion by retraining only the denoise encoder.","Discriminative audioencoders are a poor foundation for vocoder-based enhancement because their embeddings lose speaker identity, so future model selection should prioritize invertible or generative representations."],"supporting_citations":[{"why":"Provides the frozen generative audioencoder whose embeddings the system denoises and from which the vocoder synthesizes.","marker":"[10]"},{"why":"Supplies the vocoder architecture and training recipe used to synthesize speech from clean and denoised embeddings.","marker":"[12]"},{"why":"Supplies one of the discriminative audioencoders compared against the generative encoder in the main evaluation.","marker":"[2]"},{"why":"Supplies the other discriminative audioencoder baseline whose speaker fidelity is shown to be much lower.","marker":"[9]"},{"why":"Provides the waveform-to-waveform enhancement model used as the subjective listening baseline.","marker":"[26]"},{"why":"Provides the Vision Transformer architecture on which the denoise encoder is based.","marker":"[11]"},{"why":"Supplies the Valentini noisy/clean test set used for intrusive, non-intrusive, and speaker-fidelity metrics.","marker":"[19]"},{"why":"Supplies the DNS1 test set used as the second evaluation benchmark.","marker":"[18]"},{"why":"Supplies the speaker-embedding model used to compute speaker-fidelity cosine similarity.","marker":"[24]"}],"fun_headline_variants":["Generative embeddings: clean speech with a tiny denoiser and vocoder","Denoise embeddings from a frozen audioencoder, then vocode: efficient SE","Small network on generative audio embeddings beats Demucs in MOS","Speaker-preserving SE via generative embeddings, not waveform masking","Efficient SE: denoise 768-D embeddings, outperform discriminative encoders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the 768-dimensional generative audioencoder embeddings retain enough speaker and timbre information that a vocoder trained only on clean embeddings can faithfully reconstruct the original speaker from a denoised noisy embedding, and that Euclidean-distance denoising removes noise without erasing that identity.","fun_headline_variants_meta":{"raw":{"variants":["Generative embeddings: clean speech with a tiny denoiser and vocoder","Denoise embeddings from a frozen audioencoder, then vocode: efficient SE","Small network on generative audio embeddings beats Demucs in MOS","Speaker-preserving SE via generative embeddings, not waveform masking","Efficient SE: denoise 768-D embeddings, outperform discriminative encoders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1779,"prompt_tokens":859,"completion_tokens":920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":475,"tokens_out":920,"duration_ms":9085,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:28.001122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single experiment would settle the identity-preservation claim: take a set of unseen speakers, enhance their noisy speech with the full Dasheng pipeline, and run speaker verification between each clean utterance and its enhanced version; if equal-error rates are far worse than clean-versus-clean rates, the pipeline is altering speaker identity rather than preserving it.","supporting_citations":[{"cited_title":"Hifi-gan-2: Studio-quality speech enhancement via generative adversarial networks condi- tioned on acoustic features,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen generative audioencoder whose embeddings the system denoises and from which the vocoder synthesizes."},{"cited_title":"Speech enhancement using self-supervised pre-trained model and vector quantization,","cited_arxiv_id":null,"evidence_quote":"Supplies the vocoder architecture and training recipe used to synthesize speech from clean and denoised embeddings."},{"cited_title":"Initially, the noisy speech is in- put into a pre-trained audioencoder, yielding a noisy embed- ding","cited_arxiv_id":null,"evidence_quote":"Supplies one of the discriminative audioencoders compared against the generative encoder in the main evaluation."},{"cited_title":"Boosting self-supervised embeddings for speech en- hancement,","cited_arxiv_id":null,"evidence_quote":"Supplies the other discriminative audioencoder baseline whose speaker fidelity is shown to be much lower."},{"cited_title":"Perceptual eval- uation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,","cited_arxiv_id":null,"evidence_quote":"Provides the waveform-to-waveform enhancement model used as the subjective listening baseline."},{"cited_title":"Self- supervised learning for speech enhancement through synthesis,","cited_arxiv_id":null,"evidence_quote":"Provides the Vision Transformer architecture on which the denoise encoder is based."},{"cited_title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the Valentini noisy/clean test set used for intrusive, non-intrusive, and speaker-fidelity metrics."},{"cited_title":"A convnet for the 2020s,","cited_arxiv_id":null,"evidence_quote":"Supplies the DNS1 test set used as the second evaluation benchmark."}],"review_version":1}