{"id":"30993b92-7767-4e8b-a329-d73e749cf9f8","arxiv_id":"2505.13843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hierarchical speech enhancement pipeline that estimates semantic tokens first and acoustic tokens second, via a factorized codec and diffusion, improves DNSMOS and downstream TTS speaker similarity in noisy far-field conditions.","lead":"This paper proposes a speech enhancement method that first recovers the semantic content of speech, then adds acoustic details, using a factorized codec and a two-stage diffusion model. The authors report quality gains over prior enhancement systems and better speaker similarity when the enhanced speech is used as a prompt for a text-to-speech system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Downstream TTS claim is confounded by shared codec: SIM-O gains may reflect codec-native signal matching, not better enhancement; an external-TTS re-evaluation is needed.","rationale":"I read the paper in good faith. The strongest claim has two parts: DNSMOS speech-quality gains and downstream TTS SIM-O gains. The SE part is plausible in direction: SISE and SISE-w/o-dis consistently exceed the conventional baselines in Table 1, DNSMOS P.835 is a standard non-intrusive metric, and the margins are reasonably large. However, no code, audio, or confidence intervals are provided, so independent confirmation is still required. The weaker part is Table 2. Because the TTS model uses the proposed factorized codec, the comparison is biased in SISE's favor: SISE output is codec-native, while the baseline outputs are not. This is a correctness risk, not merely a disagreement with consensus. The reader's named weakest assumption (Sec. 2.1 disentanglement) is also not quantitatively established: the only ablation, SISE-w/o-dis, changes both the codec and the diffusion hierarchy, so it cannot isolate the semantic/acoustic separation, and the paper admits only 'preliminary experiments' on this point. I therefore partially agree with the reader. The TTS confound is the more decisive threat to the central claim as stated, and one external-TTS re-run would settle it. The verdict should remain conditional: the SE quality claim could survive with released artifacts, but the downstream TTS claim should not be accepted without the external-TTS check.","tokens_in":9448,"tokens_out":6823,"duration_ms":65909,"concrete_test":"Re-run the Table 2 downstream evaluation with a fixed external TTS platform whose codec is not the proposed one—for example, the released SeedTTS or MaskGCT model, or an open zero-shot TTS with frozen codec—using the same enhanced prompts and identical ASR/WavLM SIM-O scoring, and report utterance-level CIs or paired significance tests. If SISE's SIM-O advantage over FRCRN and MOSS-FORMER disappears or reverses, the downstream TTS claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the abstract's claim that SISE 'enhances TTS performance in noisy environments' is that the SIM-O gains in Table 2 come from better enhancement rather than from the TTS evaluation sharing the proposed codec. In the Experimental Setup, the zero-shot TTS model is 'similar to MaskGCT [34], except that the speech semantic representation codec and speech acoustic codec in [34] are replaced by the proposed codec'. SISE's enhanced speech is generated from tokens of this codec and decoded by it, so it lies in the codec's native reconstruction manifold. Noisy, FRCRN, and MOSS-FORMER prompts are external waveforms that must be re-encoded by the same codec, potentially suffering out-of-domain codec artifacts. The test-en SIM-O jump from 0.15/0.16 to 0.35 (Table 2) could therefore reflect codec/pipeline matching rather than restored speaker information, and no confidence intervals or significance tests are reported. The DNSMOS SE results in Table 1 are not invalidated by this concern, but the downstream-TTS half of the central claim is underdetermined as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SISE, a speech enhancement method that first estimates a semantic token stream from noisy speech and then estimates an acoustic token stream conditioned on it, using a factorized neural codec and masked diffusion models. The codec is trained with an 8-dimensional VQ bottleneck and a phoneme-prediction loss to separate semantic and acoustic attributes; the diffusion models generate the semantic token stream and the five-layer acoustic token stream hierarchically. Experiments on DNS 2021 and SeedTTS test sets compare against DCCRN, FRCRN, and MOSS-FORMER in DNSMOS P.835, and evaluate enhanced speech as prompts in a zero-shot TTS system. The paper reports the best BAK/SIG/PMOS on all three test sets and improved TTS SIM-O (e.g., 0.35 vs 0.15-0.16 on test-en), concluding that hierarchical semantic modeling improves SE and downstream TTS.","tokens_in":9695,"tokens_out":7237,"duration_ms":65235,"significance":"If the claims held, the paper would make a useful contribution: it demonstrates a concrete way to inject semantic structure into generative SE and provides a large-scale comparison with three strong baselines and a downstream TTS evaluation. The method is described with enough detail to reproduce its training schedule and inference sampling hyperparameters. However, the current evaluation does not cleanly separate the contribution of semantic factorization from the choice of codec, and the downstream TTS comparison is confounded by shared codec use. The evidence for the central mechanism is therefore not yet established, and the reported gains on DNS 2021 are modest without significance testing.","major_comments":[{"comment":"The downstream TTS evaluation is confounded by the use of the proposed codec inside the TTS model. Because the TTS model replaces the MaskGCT codecs with the proposed codec, SISE's enhanced prompts are generated and decoded within that codec's reconstruction manifold, whereas noisy, FRCRN, and MOSS-FORMER prompts are external waveforms that must be encoded by the codec at inference time. The large SIM-O gains (0.15 to 0.35 on test-en; 0.29 to 0.53 on test-zh) may therefore reflect codec/pipeline matching rather than restored speaker information. Please re-run the prompt evaluation with a fixed external TTS system (e.g., the original MaskGCT or SeedTTS), or at least with all prompts passed through the proposed codec's encoder-decoder, and report confidence intervals.","section":"Section 3, Table 2"},{"comment":"The ablation SISE-w/o-dis simultaneously changes the codec (DAC vs proposed factorized codec) and the diffusion structure (single-stream all-token prediction vs hierarchical semantic-then-acoustic prediction). Thus the PMOS and SIM-O differences between SISE-w/o-dis and SISE cannot be attributed to semantic disentanglement; they could be caused by codec reconstruction quality or tokenization. Please add an ablation that uses the proposed codec with a single diffusion that predicts all six token layers jointly, keeping everything else fixed.","section":"Section 3, Experimental Results, Tables 1 and 2"},{"comment":"The disentanglement claim is supported only by 'preliminary experiments' with no quantitative results. The 8-dimensional VQ bottleneck and phoneme-prediction loss are asserted to separate semantic and acoustic attributes, but no evidence shows that Z_s is noise-robust, that Z_a contains no semantic information, or that the bottleneck dimension is not simply trading off reconstruction fidelity. Please provide quantitative disentanglement diagnostics (e.g., phoneme prediction from Z_s versus Z_a, reconstruction quality as a function of bottleneck dimension, an ablation without L_sem) and a noise-robustness analysis of the semantic token stream.","section":"Section 2.1"},{"comment":"It is not clear whether the acoustic diffusion is trained with the clean semantic token sequence Z_s or the estimated sequence \\hat{Z}_s from the semantic diffusion. The introduction says 'estimated semantic attribute' is used as a condition, but Eq. (2) and Fig. 2 write Z_s. If training uses ground-truth Z_s while inference uses \\hat{Z}_s, the conditional distribution is mismatched and the reported gains may partly reflect oracle conditioning. Please state the exact training condition and, if clean tokens are used, describe any scheduled-sampling or noise-injection strategy.","section":"Section 2.2, Eq. (2), Fig. 2"},{"comment":"The DNS 2021 PMOS differences over the strongest baseline are small (3.58 vs 3.50 for FRCRN), and no confidence intervals or significance tests are reported for any metric. Given that DNSMOS is a non-intrusive proxy with utterance-level variance, please report 95% confidence intervals or per-utterance significance tests (e.g., paired bootstrap) for Tables 1 and 2, or specify a prespecified effect size.","section":"Table 1"}],"minor_comments":[{"comment":"In the text the SeedTTS Chinese set is called 'test-cn', while Table 1 lists it as 'test-zh'; please make the naming consistent.","section":"Section 3, Experimental Results"},{"comment":"L_sem is not defined; specify the phoneme predictor's loss, label granularity, and how the labels are aligned to the 80-Hz codec frame rate.","section":"Eq. (1)"},{"comment":"The manuscript does not state whether DCCRN, FRCRN, and MOSS-FORMER were retrained on the same data or used as released checkpoints; this is important because DNSMOS scores vary with training conditions.","section":"Section 3, Baselines"},{"comment":"The claim of being 'the first work to analyze and disentangle the detailed semantic and acoustic attributes of speech in SE' should be qualified in view of NaturalSpeech 3 [17], which already factorizes semantic and acoustic streams; the novelty statement should focus on applying this to SE.","section":"Introduction"},{"comment":"The hyperparameter section lists diffusion steps, top-k, and temperature, but not the total number of diffusion training steps or batch size; adding these would help reproducibility.","section":"Section 3, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within scope for a speech-processing venue. The main risk is that the downstream TTS claim is overstated because of the shared-codec confound, and the current ablation cannot support the claim that semantic factorization itself helps. I would not accept without an external-TTS re-evaluation and a cleaner ablation of the hierarchical diffusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2505.13843. What's genuinely new is the composition: a factorized codec (semantic and acoustic tokens) combined with a two-stage discrete diffusion for speech enhancement, where the semantic tokens of clean speech are predicted first from noisy speech and then used as a condition for acoustic-token prediction. The pieces come from NaturalSpeech 3, DAC, SoundStorm, and token-critic, but the hierarchy for SE is new and clearly motivated. The internal ablation supports the design: SISE beats SISE-w/o-dis, which uses the same codec base without the semantic/acoustic split, by 0.1 PMOS on SeedTTS test-en and by 0.06 SIM-O in the TTS test. The DNSMOS results on the DNS 2021 set are modest (PMOS 3.58 vs 3.50 for the best conventional baseline) but consistent across three test sets. The main soft spot is the downstream TTS claim. The zero-shot TTS model is described as 'similar to MaskGCT, except that the speech semantic representation codec and speech acoustic codec are replaced by the proposed codec.' Since SISE's output is decoded from that codec, it sits on the codec's native manifold. The FRCRN and MOSS-FORMER prompts are external waveforms that must be re-encoded by the same codec, and they may pick up out-of-domain artifacts. So the SIM-O jump from 0.15/0.16 to 0.35 could reflect codec/pipeline matching, not better enhancement. The same confound applies to the test-zh results. The paper's own phrasing—'this modification also results in a competitive TTS framework'—doesn't resolve the issue. I'd want to see the TTS evaluation redone with a fixed external TTS (e.g., the original MaskGCT) and with confidence intervals or significance tests. Two more minor issues: the DNSMOS differences are small on the DNS set, and no uncertainty is reported; and the disentanglement claim rests on 'preliminary experiments' with no quantitative analysis of semantic/acoustic separation or noise robustness. No code or audio is provided, which limits independent checking. Who is this for? People working on codec-based speech enhancement or robust TTS front-ends. It deserves a serious referee, but with major revision: external-TTS re-evaluation, error bars, and ideally released samples. I'd accept it for review on the strength of the architecture and the internal ablation; I just wouldn't take the TTS claim at face value as written.","headline":"A plausible hierarchical codec-diffusion SE pipeline with a real confound in the downstream TTS evaluation.","tokens_in":10235,"tokens_out":2620,"would_cite":false,"duration_ms":22694,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic-first token prediction beats one-shot speech enhancement.","keywords":["speech enhancement","factorized codec","semantic tokens","acoustic tokens","discrete diffusion","zero-shot text-to-speech","speaker similarity","far-field noisy speech"],"falsifier":"Train the same codec and diffusion stack but remove the semantic conditioning, predicting all token layers jointly from the noisy encoding alone, and compare on the same far-field test sets; if its background, signal, and overall quality scores and downstream speaker similarity match the proposed method, then the step-by-step semantic-first hierarchy is not what carries the gain. A second check would be to corrupt semantic tokens or acoustic tokens of clean speech separately and measure how much each corruption changes the phoneme content and speaker identity of the decoded audio, since the claimed disentanglement predicts that semantic corruption destroys content while acoustic corruption mostly changes timbre.","tokens_in":9242,"feed_emoji":"🎧","tokens_out":9381,"duration_ms":79214,"temperature":0.7,"pith_summary":"This paper tries to establish that a noisy speech signal is best cleaned not by estimating the clean waveform or spectrum in one step, but by first recovering a small semantic token stream that carries the linguistic content, then recovering an acoustic token stream conditioned on it, and finally decoding the two streams back into speech. The paper builds a factorized codec whose discrete token space is split into semantic and acoustic parts by an information bottleneck and a phoneme-prediction loss, then trains two masked-token diffusion models to estimate the clean tokens from the noisy input, step by step. On the standard far-field noisy test set and on English and Chinese test sets with added noise and room impulse responses, the system reports the highest background, signal, and overall quality scores among the compared methods, and it more than doubles the speaker similarity of zero-shot TTS prompts relative to the strongest conventional baseline. The reason this matters is that if the semantic stream is easier to estimate than the full signal in far-field, low-SNR conditions, then converting enhancement into hierarchical generation could make front-ends for hearing aids, ASR, and TTS more robust.","feed_headline":"Semantic-first token prediction beats one-shot speech enhancement","feed_subtitle":"A factorized codec plus two diffusion steps raises noisy-prompt speaker similarity from 0.15 to 0.35.","key_machinery":"The machinery is a factorized codec paired with two masked-token diffusion models. The codec writes speech into two codebooks: a one-layer semantic codebook pushed by an 8-dimensional information bottleneck and a frame-level phoneme-prediction loss, and a five-layer acoustic codebook that carries the remaining detail; summing the two quantized streams and decoding reconstructs clean speech. The semantic diffusion fills in masked semantic tokens using only the noisy encoding as condition, and the acoustic diffusion fills in masked acoustic tokens layer by layer using the noisy encoding plus the estimated semantic tokens, with a masking schedule and confidence-based remasking. This decomposition does the work: the semantic stream is a smaller, content-dominated target that is easier to estimate in low-SNR conditions, and once it is known it narrows what the acoustic model has to guess.","core_discovery":"The central claim is that semantic and acoustic attributes of speech can be disentangled into two discrete token streams, and that predicting the semantic stream first from noisy speech and the acoustic stream second, conditioned on the estimated semantic tokens, yields cleaner recovered speech than estimating all tokens or masking spectrograms directly. Concretely, the codec projects its encoder output into an 8-dimensional space before vector quantization, which acts as an information bottleneck, and adds a phoneme-prediction loss on the semantic tokens to force content-related information into that stream. The semantic diffusion operates on a single token layer with only the noisy speech as condition; the acoustic diffusion then operates on five residual token layers with the noisy encoding and the estimated semantic tokens as condition, using a masking schedule and confidence-based remasking. On the standard far-field noisy test set and on English and Chinese test sets with added noise and room impulse responses, the method reports the best background, signal, and overall quality scores among the compared systems, and in zero-shot text-to-speech the speaker similarity of speech generated from enhanced prompts rises to 0.35 on the English set, compared with 0.15 for the best conventional baseline and 0.54 for clean ground-truth prompts.","pith_inferences":["If the semantic bottleneck is what makes estimation easier, coupling this pipeline with an external language model or ASR hypotheses as an additional condition could push enhancement further at very low SNR, since the semantic stream is phoneme-aligned; the paper does not test this.","The same semantic token stream could be read directly by a downstream recognizer instead of only by a TTS decoder, potentially making robust ASR an explicit product of the enhancement step; the paper evaluates TTS but not ASR.","Because the codec relies on forced phoneme alignment, the method is limited to languages and data with aligners available; a self-supervised semantic target would be needed to extend it, which the paper leaves open.","The reported sampling budget, fifteen semantic steps and ten plus three single steps over acoustic layers, implies a latency-reduction experiment: distilling the diffusion to fewer steps and measuring the quality-versus-speed trade-off, which the paper does not report."],"forward_implications":["If the hierarchy is the cause of the gains, enhancement systems can be redesigned as semantic-first generative pipelines instead of one-shot mask or spectrum estimators.","The same factorized codec and diffusion pair can serve as a front-end for zero-shot text-to-speech, raising speaker similarity of generated speech from noisy prompts without degrading word error rate.","The evaluation on both English and Chinese test sets indicates the benefit is not confined to a single language, within the tested conditions.","The method also carries over to other speech restoration tasks that share the clean-speech recovery goal, such as declipping and bandwidth extension.","The reported quality advantage over the conventional baselines is largest in the far-field, low-SNR conditions, which is where the paper argues semantic estimation helps most."],"supporting_citations":[{"why":"supplies the semantic/acoustic factorization idea, the information-bottleneck projection, and the phoneme-supervision objective that the codec adapts","marker":"[17]"},{"why":"provides the codec architecture, residual vector quantization, discriminators, and reconstruction losses used to train the semantic-based codec","marker":"[20]"},{"why":"provides the masked-token diffusion formulation, including the masking schedule and confidence-based remasking used by both diffusion models","marker":"[18]"},{"why":"supplies the layer-by-layer acoustic token generation and conditioning scheme that the acoustic diffusion follows","marker":"[23]"},{"why":"provides the noise and room-impulse-response corpora and the mixing script used to build the far-field noisy training and evaluation data","marker":"[26]"},{"why":"supplies the non-intrusive perceptual metric that produces the reported BAK, SIG, and PMOS quality scores","marker":"[30]"},{"why":"a state-of-the-art enhancement baseline that the proposed method must beat in the quality comparisons","marker":"[10]"},{"why":"a second state-of-the-art enhancement baseline used in both quality and downstream TTS comparisons","marker":"[11]"},{"why":"provides the zero-shot text-to-speech architecture that is retrained with the proposed codec and used to evaluate enhanced prompts","marker":"[34]"},{"why":"provides the English and Chinese clean speech evaluation sets that are mixed with noise and room impulse responses for the far-field tests","marker":"[27]"}],"fun_headline_variants":["Semantic-first token prediction beats one-shot speech enhancement","Factorized codec plus dual diffusion lifts speech quality","Semantic then acoustic diffusion cleans speech and TTS","Hierarchical codec boosts speaker similarity to 0.35","Predict semantic tokens first for robust speech recovery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that the codec's bottleneck and phoneme-supervision actually split speech into a noise-robust semantic stream and a residual acoustic stream, so that estimating the semantic stream from noise is reliably easier than estimating the full signal; the paper reports only preliminary experiments for that separation, with no quantitative measurement of how cleanly the two streams are disentangled or how noise affects each one.","fun_headline_variants_meta":{"raw":{"variants":["Semantic-first token prediction beats one-shot speech enhancement","Factorized codec plus dual diffusion lifts speech quality","Semantic then acoustic diffusion cleans speech and TTS","Hierarchical codec boosts speaker similarity to 0.35","Predict semantic tokens first for robust speech recovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1759,"prompt_tokens":932,"completion_tokens":827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":751}},"tokens_in":548,"tokens_out":827,"duration_ms":6790,"temperature":1.0,"reasoning_tokens":751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:08:37.928208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same codec and diffusion stack but remove the semantic conditioning, predicting all token layers jointly from the noisy encoding alone, and compare on the same far-field test sets; if its background, signal, and overall quality scores and downstream speaker similarity match the proposed method, then the step-by-step semantic-first hierarchy is not what carries the gain. A second check would be to corrupt semantic tokens or acoustic tokens of clean speech separately and measure how much each corruption changes the phoneme content and speaker identity of the decoded audio, since the claimed disentanglement predicts that semantic corruption destroys content while acoustic corruption mostly changes timbre.","supporting_citations":[{"cited_title":"A deep represen- tation learning-based speech enhancement method using complex convolution recurrent variational autoencoder,","cited_arxiv_id":null,"evidence_quote":"supplies the semantic/acoustic factorization idea, the information-bottleneck projection, and the phoneme-supervision objective that the codec adapts"},{"cited_title":"Low-latency speech enhancement via speech token generation,","cited_arxiv_id":null,"evidence_quote":"provides the codec architecture, residual vector quantization, discriminators, and reconstruction losses used to train the semantic-based codec"},{"cited_title":"Unsupervised speech decomposition via triple information bot- tleneck,","cited_arxiv_id":null,"evidence_quote":"supplies the layer-by-layer acoustic token generation and conditioning scheme that the acoustic diffusion follows"},{"cited_title":"Inter- speech 2021 deep noise suppression challenge,","cited_arxiv_id":null,"evidence_quote":"supplies the non-intrusive perceptual metric that produces the reported BAK, SIG, and PMOS quality scores"},{"cited_title":"Conv-tasnet: Surpassing ideal time– frequency magnitude masking for speech separation,","cited_arxiv_id":null,"evidence_quote":"a state-of-the-art enhancement baseline that the proposed method must beat in the quality comparisons"},{"cited_title":"A parallel-data-free speech enhancement method using multi-objective learning cycle-consistent generative adversarial network,","cited_arxiv_id":null,"evidence_quote":"a second state-of-the-art enhancement baseline used in both quality and downstream TTS comparisons"},{"cited_title":"DNSMOS P.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,","cited_arxiv_id":null,"evidence_quote":"provides the zero-shot text-to-speech architecture that is retrained with the proposed codec and used to evaluate enhanced prompts"}],"review_version":1}