Pith. sign in

REVIEW 5 major objections 5 minor 39 references

A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Semantic-first token prediction beats one-shot speech enhancement.

desk verdict A plausible hierarchical codec-diffusion SE pipeline with a real confound in the downstream TTS evaluation. read the letter →

arxiv 2505.13843 v1 pith:SA54XXT5 submitted 2025-05-20 eess.AS cs.SD

Yang Xiang , Canan Huang , Desheng Hu , Jingguang Tian , Xinhui Hu , Chao Zhang This is my paper · ORCID
classification eess.AScs.SD
keywords speechenhancementfactorizedcodecsemantictokensacousticdiscretediffusionzero-shottext-to-speechspeakersimilarityfar-fieldnoisy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a noisy speech signal is best cleaned not by estimating the clean waveform or spectrum in one step, but by first recovering a small semantic token stream that carries the linguistic content, then recovering an acoustic token stream conditioned on it, and finally decoding the two streams back into speech. The paper builds a factorized codec whose discrete token space is split into semantic and acoustic parts by an information bottleneck and a phoneme-prediction loss, then trains two masked-token diffusion models to estimate the clean tokens from the noisy input, step by step. On the standard far-field noisy test set and on English and Chinese test sets with added noise and room impulse responses, the system reports the highest background, signal, and overall quality scores among the compared methods, and it more than doubles the speaker similarity of zero-shot TTS prompts relative to the strongest conventional baseline. The reason this matters is that if the semantic stream is easier to estimate than the full signal in far-field, low-SNR conditions, then converting enhancement into hierarchical generation could make front-ends for hearing aids, ASR, and TTS more robust.

What carries the argument

The machinery is a factorized codec paired with two masked-token diffusion models. The codec writes speech into two codebooks: a one-layer semantic codebook pushed by an 8-dimensional information bottleneck and a frame-level phoneme-prediction loss, and a five-layer acoustic codebook that carries the remaining detail; summing the two quantized streams and decoding reconstructs clean speech. The semantic diffusion fills in masked semantic tokens using only the noisy encoding as condition, and the acoustic diffusion fills in masked acoustic tokens layer by layer using the noisy encoding plus the estimated semantic tokens, with a masking schedule and confidence-based remasking. This decomposition does the work: the semantic stream is a smaller, content-dominated target that is easier to estimate in low-SNR conditions, and once it is known it narrows what the acoustic model has to guess.

What would settle it

Train the same codec and diffusion stack but remove the semantic conditioning, predicting all token layers jointly from the noisy encoding alone, and compare on the same far-field test sets; if its background, signal, and overall quality scores and downstream speaker similarity match the proposed method, then the step-by-step semantic-first hierarchy is not what carries the gain. A second check would be to corrupt semantic tokens or acoustic tokens of clean speech separately and measure how much each corruption changes the phoneme content and speaker identity of the decoded audio, since the claimed disentanglement predicts that semantic corruption destroys content while acoustic corruption mostly changes timbre.

Watch

Extended reading notes

Core claim

The central claim is that semantic and acoustic attributes of speech can be disentangled into two discrete token streams, and that predicting the semantic stream first from noisy speech and the acoustic stream second, conditioned on the estimated semantic tokens, yields cleaner recovered speech than estimating all tokens or masking spectrograms directly. Concretely, the codec projects its encoder output into an 8-dimensional space before vector quantization, which acts as an information bottleneck, and adds a phoneme-prediction loss on the semantic tokens to force content-related information into that stream. The semantic diffusion operates on a single token layer with only the noisy speech as condition; the acoustic diffusion then operates on five residual token layers with the noisy encoding and the estimated semantic tokens as condition, using a masking schedule and confidence-based remasking. On the standard far-field noisy test set and on English and Chinese test sets with added noise and room impulse responses, the method reports the best background, signal, and overall quality scores among the compared systems, and in zero-shot text-to-speech the speaker similarity of speech generated from enhanced prompts rises to 0.35 on the English set, compared with 0.15 for the best conventional baseline and 0.54 for clean ground-truth prompts.

Load-bearing premise

The method rests on the assumption that the codec's bottleneck and phoneme-supervision actually split speech into a noise-robust semantic stream and a residual acoustic stream, so that estimating the semantic stream from noise is reliably easier than estimating the full signal; the paper reports only preliminary experiments for that separation, with no quantitative measurement of how cleanly the two streams are disentangled or how noise affects each one.

Editorial extensions

If this is right

  • If the hierarchy is the cause of the gains, enhancement systems can be redesigned as semantic-first generative pipelines instead of one-shot mask or spectrum estimators.
  • The same factorized codec and diffusion pair can serve as a front-end for zero-shot text-to-speech, raising speaker similarity of generated speech from noisy prompts without degrading word error rate.
  • The evaluation on both English and Chinese test sets indicates the benefit is not confined to a single language, within the tested conditions.
  • The method also carries over to other speech restoration tasks that share the clean-speech recovery goal, such as declipping and bandwidth extension.
  • The reported quality advantage over the conventional baselines is largest in the far-field, low-SNR conditions, which is where the paper argues semantic estimation helps most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the semantic bottleneck is what makes estimation easier, coupling this pipeline with an external language model or ASR hypotheses as an additional condition could push enhancement further at very low SNR, since the semantic stream is phoneme-aligned; the paper does not test this.
  • The same semantic token stream could be read directly by a downstream recognizer instead of only by a TTS decoder, potentially making robust ASR an explicit product of the enhancement step; the paper evaluates TTS but not ASR.
  • Because the codec relies on forced phoneme alignment, the method is limited to languages and data with aligners available; a self-supervised semantic target would be needed to extend it, which the paper leaves open.
  • The reported sampling budget, fifteen semantic steps and ten plus three single steps over acoustic layers, implies a latency-reduction experiment: distilling the diffusion to fewer steps and measuring the quality-versus-speed trade-off, which the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes SISE, a speech enhancement method that first estimates a semantic token stream from noisy speech and then estimates an acoustic token stream conditioned on it, using a factorized neural codec and masked diffusion models. The codec is trained with an 8-dimensional VQ bottleneck and a phoneme-prediction loss to separate semantic and acoustic attributes; the diffusion models generate the semantic token stream and the five-layer acoustic token stream hierarchically. Experiments on DNS 2021 and SeedTTS test sets compare against DCCRN, FRCRN, and MOSS-FORMER in DNSMOS P.835, and evaluate enhanced speech as prompts in a zero-shot TTS system. The paper reports the best BAK/SIG/PMOS on all three test sets and improved TTS SIM-O (e.g., 0.35 vs 0.15-0.16 on test-en), concluding that hierarchical semantic modeling improves SE and downstream TTS.

Significance. If the claims held, the paper would make a useful contribution: it demonstrates a concrete way to inject semantic structure into generative SE and provides a large-scale comparison with three strong baselines and a downstream TTS evaluation. The method is described with enough detail to reproduce its training schedule and inference sampling hyperparameters. However, the current evaluation does not cleanly separate the contribution of semantic factorization from the choice of codec, and the downstream TTS comparison is confounded by shared codec use. The evidence for the central mechanism is therefore not yet established, and the reported gains on DNS 2021 are modest without significance testing.

major comments (5)
  1. [Section 3, Table 2] The downstream TTS evaluation is confounded by the use of the proposed codec inside the TTS model. Because the TTS model replaces the MaskGCT codecs with the proposed codec, SISE's enhanced prompts are generated and decoded within that codec's reconstruction manifold, whereas noisy, FRCRN, and MOSS-FORMER prompts are external waveforms that must be encoded by the codec at inference time. The large SIM-O gains (0.15 to 0.35 on test-en; 0.29 to 0.53 on test-zh) may therefore reflect codec/pipeline matching rather than restored speaker information. Please re-run the prompt evaluation with a fixed external TTS system (e.g., the original MaskGCT or SeedTTS), or at least with all prompts passed through the proposed codec's encoder-decoder, and report confidence intervals.
  2. [Section 3, Experimental Results, Tables 1 and 2] The ablation SISE-w/o-dis simultaneously changes the codec (DAC vs proposed factorized codec) and the diffusion structure (single-stream all-token prediction vs hierarchical semantic-then-acoustic prediction). Thus the PMOS and SIM-O differences between SISE-w/o-dis and SISE cannot be attributed to semantic disentanglement; they could be caused by codec reconstruction quality or tokenization. Please add an ablation that uses the proposed codec with a single diffusion that predicts all six token layers jointly, keeping everything else fixed.
  3. [Section 2.1] The disentanglement claim is supported only by 'preliminary experiments' with no quantitative results. The 8-dimensional VQ bottleneck and phoneme-prediction loss are asserted to separate semantic and acoustic attributes, but no evidence shows that Z_s is noise-robust, that Z_a contains no semantic information, or that the bottleneck dimension is not simply trading off reconstruction fidelity. Please provide quantitative disentanglement diagnostics (e.g., phoneme prediction from Z_s versus Z_a, reconstruction quality as a function of bottleneck dimension, an ablation without L_sem) and a noise-robustness analysis of the semantic token stream.
  4. [Section 2.2, Eq. (2), Fig. 2] It is not clear whether the acoustic diffusion is trained with the clean semantic token sequence Z_s or the estimated sequence \hat{Z}_s from the semantic diffusion. The introduction says 'estimated semantic attribute' is used as a condition, but Eq. (2) and Fig. 2 write Z_s. If training uses ground-truth Z_s while inference uses \hat{Z}_s, the conditional distribution is mismatched and the reported gains may partly reflect oracle conditioning. Please state the exact training condition and, if clean tokens are used, describe any scheduled-sampling or noise-injection strategy.
  5. [Table 1] The DNS 2021 PMOS differences over the strongest baseline are small (3.58 vs 3.50 for FRCRN), and no confidence intervals or significance tests are reported for any metric. Given that DNSMOS is a non-intrusive proxy with utterance-level variance, please report 95% confidence intervals or per-utterance significance tests (e.g., paired bootstrap) for Tables 1 and 2, or specify a prespecified effect size.
minor comments (5)
  1. [Section 3, Experimental Results] In the text the SeedTTS Chinese set is called 'test-cn', while Table 1 lists it as 'test-zh'; please make the naming consistent.
  2. [Eq. (1)] L_sem is not defined; specify the phoneme predictor's loss, label granularity, and how the labels are aligned to the 80-Hz codec frame rate.
  3. [Section 3, Baselines] The manuscript does not state whether DCCRN, FRCRN, and MOSS-FORMER were retrained on the same data or used as released checkpoints; this is important because DNSMOS scores vary with training conditions.
  4. [Introduction] The claim of being 'the first work to analyze and disentangle the detailed semantic and acoustic attributes of speech in SE' should be qualified in view of NaturalSpeech 3 [17], which already factorizes semantic and acoustic streams; the novelty statement should focus on applying this to SE.
  5. [Section 3, Implementation Details] The hyperparameter section lists diffusion steps, top-k, and temperature, but not the total number of diffusion training steps or batch size; adding these would help reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

Independent external metrics (DNSMOS, WER, SIM-O) and an internal codec-matched ablation support SISE's central claims; the only notable caveat is the shared codec in the downstream TTS evaluation, which is a generalization concern rather than a circular derivation, so no circular steps are found.

full rationale

The claimed derivation chain is self-contained. SISE first trains a factorized codec on clean speech with an information bottleneck and a phoneme prediction loss (Sec. 2.1, Eq. 1), then trains a semantic diffusion that maps noisy speech to clean semantic tokens and an acoustic diffusion that maps noisy speech plus estimated semantic tokens to acoustic tokens (Sec. 2.2, Eqs. 2-3). No equation defines the predicted tokens in terms of the enhancement output or vice versa; the codec is trained on clean speech, and the diffusion targets are clean tokens generated by that codec, with noisy speech only as a conditioning input. The speech-quality claim is scored with the external DNSMOS P.835 metric, and the TTS metrics use external Whisper/Paraformer and WavLM, so the central quality comparison does not reduce to the method's own components. Architecture choices follow external prior work (DAC, NaturalSpeech 3, SoundStorm, MaskGCT), and the paper's self-citations [4,7,12,13] are background references, not load-bearing evidence. Two caveats are noted but they are not circularity. First, the paper says 'Based on our preliminary experiments, the information bottleneck and supervised learning are sufficient for effective disentanglement' (Sec. 2.1) without reporting those experiments; this is a missing-support concern, not a circular step. Second, the downstream zero-shot TTS model is described as 'similar to MaskGCT [34], except that the speech semantic representation codec and speech acoustic codec in [34] are replaced by the proposed codec,' so SISE's enhanced output is in-distribution for the TTS front-end while waveform baselines must be re-encoded; this can inflate the absolute SIM-O gap, but the internal comparison SISE vs SISE-w/o-dis uses codec-based enhancement on both sides and still shows a gain, so the hierarchical modeling claim is not forced by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The ledger shows the paper's contribution rests on a set of domain assumptions about token disentanglement and on manually chosen hyperparameters. No new physical entities are introduced.

free parameters (5)
  • loss weights for codec training = lambda_rec=5, lambda_adv=4, lambda_feat=4, lambda_codebook=1, lambda_commit=1, lambda_sem=10
    Set without reported ablations; the semantic loss weight (10) is central to the disentanglement claim.
  • VQ information bottleneck dimension = 8
    Projection to 8 dims before quantization; claimed to force disentanglement, no ablation varying dimensionality.
  • number of VQ layers = 1 semantic layer, 5 acoustic layers
    Architecture choice inherited from prior codec design, determines token granularity.
  • diffusion inference steps = semantic: 15, acoustic: [10,1,1,1,1]
    Chosen for quality/speed; no sensitivity analysis.
  • sampling parameters = top-k=20, temperature anneal 1.5 to 0, Gumbel noise
    Sampling strategy from MaskGCT; not tuned for this use case in the paper.
assumptions (6)
  • domain assumption Speech can be factorized into independent semantic and acoustic token streams such that predicting the semantic stream first improves enhancement.
    Core hypothesis of SISE, stated in Sec. 1 and Sec. 2.1; not derived.
  • domain assumption The 8-dim information bottleneck plus phoneme supervision yields the intended disentanglement.
    Sec. 2.1: 'Based on our preliminary experiments, the information bottleneck and supervised learning are sufficient.' No quantitative disentanglement evaluation.
  • standard math Masked-token diffusion with confidence-based remasking is a valid generative model for speech tokens.
    Adopted from token-critic [18], SoundStorm [23], and MaskGCT [34]; the paper relies on these as proven.
  • domain assumption DNSMOS P.835 is an adequate proxy for human perception of enhanced speech.
    Used as sole quality metric; no listening test. DNSMOS is known to correlate with human ratings but is not exact.
  • domain assumption MFA frame-level phoneme labels on 100K hours of clean Emilia speech are accurate enough to supervise semantic token learning.
    MFA alignments on large multilingual data can contain errors; no verification.
  • domain assumption The noisy encoder output at downsample rate 200 preserves enough information to condition token generation.
    Used for both semantic and acoustic diffusion as condition; no analysis of information retained.
invented entities (2)
  • Semantic token stream Z_s
    purpose: Discrete, phoneme-related token that is predicted first from noisy speech, then used as a condition for acoustic token generation.
    The semantic content is enforced only by a phoneme prediction loss on the codec, so the token's 'semantic' property is defined by the paper's own training objective. No external task demonstrates that the token carries semantic content beyond this construction.
  • Acoustic token stream Z_a
    purpose: Discrete token capturing remaining acoustic details; generated conditioned on estimated semantic tokens and noisy speech.
    Defined as the residual after subtracting semantic tokens; no independent evidence that it specifically captures acoustic attributes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model." pith.science (2026). https://pith.science/paper/SA54XXT5

@misc{pith2026250513843,
  author       = {Pith},
  title        = {Pith review of: A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SA54XXT5}},
  note         = {Machine review of arXiv:2505.13843}
}
read the original abstract

Most current speech enhancement (SE) methods recover clean speech from noisy inputs by directly estimating time-frequency masks or spectrums. However, these approaches often neglect the distinct attributes, such as semantic content and acoustic details, inherent in speech signals, which can hinder performance in downstream tasks. Moreover, their effectiveness tends to degrade in complex acoustic environments. To overcome these challenges, we propose a novel, semantic information-based, step-by-step factorized SE method using factorized codec and diffusion model. Unlike traditional SE methods, our hierarchical modeling of semantic and acoustic attributes enables more robust clean speech recovery, particularly in challenging acoustic scenarios. Moreover, this method offers further advantages for downstream TTS tasks. Experimental results demonstrate that our algorithm not only outperforms SOTA baselines in terms of speech quality but also enhances TTS performance in noisy environments.

Figures

Figures reproduced from arXiv: 2505.13843 by the authors.

Figure 1
Figure 1. Overview of the SISE framework with a factorized codec and factorized diffusion model. The red part will be ap￾plied in both the training and inference processes, while the black part will be used only during the training stage. is denoted as y, and the clean speech signal is x. To analyze the semantic information during SE, we factorize the speech into semantic and acoustic attributes. These attributes are rep￾rese… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

  1. [1]

    SE has been widely applied in various fields, including hear- ing aids, text-to-speech (TTS) systems, and robust automatic speech recognition (ASR) [1, 2]

    Introduction The aim of speech enhancement (SE) is to remove background noise and improve the quality and intelligibility of noisy speech. SE has been widely applied in various fields, including hear- ing aids, text-to-speech (TTS) systems, and robust automatic speech recognition (ASR) [1, 2]. Over the past decades, many SE algorithms have been developed ...

    work page

  2. [2]

    A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model

    Semantic Information-Based SE In this section, we present the SISE framework. As shown in Fig.1, SISE consists of a semantic-based codec and a factorized diffusion model. In this work, we focus on achieving SE in far- field environments with complex noise. The observed signal arXiv:2505.13843v1 [eess.AS] 20 May 2025 Codec encoder Codec decoder Noisy encod...

    work page Pith review arXiv 2025

  3. [3]

    Specifically, we will first assess the enhanced speech quality in complex, far-field noisy environments

    Experimental Setup & Results This section will evaluate the SE performance of the proposed algorithm. Specifically, we will first assess the enhanced speech quality in complex, far-field noisy environments. Then, we will evaluate its performance in a zero-shot TTS task. For the zero- shot TTS task, we will enhance the noisy prompt speech to de- termine wh...

    work page 2021

  4. [4]

    Conclusions This paper presents a novel SE method that leverages seman- tic information-based factorization. Unlike traditional SE algo- rithms, our approach hierarchically models both semantic and acoustic attributes, improving the estimation of clean speech in complex acoustic environments. Experimental results show that the proposed SISE outperforms cu...

    work page

  5. [5]

    Human Listening and Live Caption- ing: Multi-Task Training for Speech Enhancement,

    S. E. Eskimez, X. Wang, M. Tang, H. Yang, Z. Zhu, Z. Chen, H. Wang, and T. Yoshioka, “Human Listening and Live Caption- ing: Multi-Task Training for Speech Enhancement,” inProc. In- terspeech, 2021, pp. 2686–2690

    work page 2021

  6. [6]

    How bad are artifacts?: Analyzing the impact of speech enhancement errors on asr,

    K. Iwamoto, T. Ochiai, M. Delcroix, R. Ikeshita, H. Sato, S. Araki, and S. Katagiri, “How bad are artifacts?: Analyzing the impact of speech enhancement errors on asr,”arXiv preprint arXiv:2201.06685, 2022

    arXiv 2022

  7. [7]

    Supervised and unsupervised speech enhancement using nonnegative matrix factorization,

    N. Mohammadiha, P. Smaragdis, and A. Leijon, “Supervised and unsupervised speech enhancement using nonnegative matrix factorization,”IEEE Trans. Audio, Speech, and Lang. Process., vol. 21, no. 10, pp. 2140–2151, 2013

    work page 2013

  8. [8]

    A novel NMF-HMM speech enhancement algo- rithm based on poisson mixture model,

    Y . Xiang, L. Shi, J. L. Højvang, M. H. Rasmussen, and M. G. Christensen, “A novel NMF-HMM speech enhancement algo- rithm based on poisson mixture model,” inProc. ICASSP, 2021, pp. 721–725

    work page 2021

Show all 39 references
  1. [9]

    Supervised speech separation based on deep learning: An overview,

    D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,”IEEE/ACM Trans. Audio, Speech, and Lang. Process., vol. 26, no. 10, pp. 1702–1726, 2018

    2018

  2. [10]

    Conv-tasnet: Surpassing ideal time– frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time– frequency magnitude masking for speech separation,”IEEE/ACM Trans. Audio, Speech, and Lang. Process., vol. 27, no. 8, pp. 1256–1266, 2019

    2019

  3. [11]

    A parallel-data-free speech enhancement method using multi-objective learning cycle-consistent generative adversarial network,

    Y . Xiang and C. Bao, “A parallel-data-free speech enhancement method using multi-objective learning cycle-consistent generative adversarial network,”IEEE/ACM Trans. Audio, Speech, and Lang. Process., vol. 28, pp. 1826–1838, 2020

    2020

  4. [12]

    DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement,

    Y . Hu, Y . Liu, S. Lv, M. Xing, S. Zhang, Y . Fu, J. Wu, B. Zhang, and L. Xie, “DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement,” inProc. Inter- speech, 2020, pp. 2472–2476

    2020

  5. [13]

    On the compensation between magnitude and phase in speech separation,

    Z.-Q. Wang, G. Wichern, and J. Le Roux, “On the compensation between magnitude and phase in speech separation,”IEEE Signal Processing Letters, vol. 28, pp. 2018–2022, 2021

    2018

  6. [14]

    FR- CRN: Boosting feature representation using frequency recurrence for monaural speech enhancement,

    S. Zhao, B. Ma, K. N. Watcharasupat, and W.-S. Gan, “FR- CRN: Boosting feature representation using frequency recurrence for monaural speech enhancement,” inProc. ICASSP, 2022, pp. 9281–9285

    2022

  7. [15]

    Mossformer2: Combining transformer and RNN-free recurrent network for enhanced time- domain monaural speech separation,

    S. Zhao, Y . Ma, C. Ni, C. Zhang, H. Wang, T. H. Nguyen, K. Zhou, J. Q. Yip, D. Ng, and B. Ma, “Mossformer2: Combining transformer and RNN-free recurrent network for enhanced time- domain monaural speech separation,” inProc. ICASSP, 2024, pp. 10 356–10 360

    2024

  8. [16]

    A two-stage deep representation learning-based speech enhancement method using variational autoencoder and adversar- ial training,

    Y . Xiang, J. L. Højvang, M. H. Rasmussen, and M. G. Chris- tensen, “A two-stage deep representation learning-based speech enhancement method using variational autoencoder and adversar- ial training,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 32, p...

    2024

  9. [17]

    A deep represen- tation learning-based speech enhancement method using complex convolution recurrent variational autoencoder,

    Y . Xiang, J. Tian, X. Hu, X. Xu, and Z. Yin, “A deep represen- tation learning-based speech enhancement method using complex convolution recurrent variational autoencoder,” inProc. ICASSP, 2024, pp. 781–785

    2024

  10. [18]

    Genhancer: High-fidelity speech enhancement via generative modeling on discrete codec tokens,

    H. Yang, J. Su, M. Kim, and Z. Jin, “Genhancer: High-fidelity speech enhancement via generative modeling on discrete codec tokens,” inProc. Interspeech, 2024, pp. 1170–1174

    2024

  11. [19]

    Diffusion models for audio restoration: A re- view [special issue on model-based and data-driven audio signal processing],

    J.-M. Lemercier, J. Richter, S. Welker, E. Moliner, V . V ¨alim¨aki, and T. Gerkmann, “Diffusion models for audio restoration: A re- view [special issue on model-based and data-driven audio signal processing],”IEEE Signal Processing Magazine, vol. 41, no. 6, pp. 72–84, 2025

    2025

  12. [20]

    Low-latency speech enhancement via speech token generation,

    H. Xue, X. Peng, and Y . Lu, “Low-latency speech enhancement via speech token generation,” inProc. ICASSP, 2024, pp. 661– 665

    2024

  13. [21]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tanget al., “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” inProc. ICML, 2024

    2024

  14. [22]

    Improved masked image generation with token-critic,

    J. Lezama, H. Chang, L. Jiang, and I. Essa, “Improved masked image generation with token-critic,” inProc. ECCV, 2022, pp. 70–86

    2022

  15. [23]

    Unsupervised speech decomposition via triple information bot- tleneck,

    K. Qian, Y . Zhang, S. Chang, M. Hasegawa-Johnson, and D. Cox, “Unsupervised speech decomposition via triple information bot- tleneck,” inProc. ICML, 2020, pp. 7836–7846

    2020

  16. [24]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    2024

  17. [25]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  18. [26]

    Neural discrete represen- tation learning,

    A. Van Den Oord, O. Vinyalset al., “Neural discrete represen- tation learning,”Advances in neural information processing sys- tems, vol. 30, 2017

    2017

  19. [27]

    Soundstorm: Efficient parallel audio gen- eration,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio gen- eration,”arXiv preprint arXiv:2305.09636, 2023

    2023 arXiv

  20. [28]

    Dif- fusion posterior sampling for general noisy inverse problems,

    H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, “Dif- fusion posterior sampling for general noisy inverse problems,” arXiv preprint arXiv:2209.14687, 2022

    2022 arXiv

  21. [29]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shiet al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” inProc. SLT, 2024, pp. 885–890

    2024

  22. [30]

    Inter- speech 2021 deep noise suppression challenge,

    C. K. Reddy, H. Dubey, K. Koishida, A. Nair, V . Gopal, R. Cut- ler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Inter- speech 2021 deep noise suppression challenge,” inProc. Inter- speech, 2021

    2021

  23. [31]

    Seed-TTS: A family of high-quality versatile speech generation models,

    P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gaoet al., “Seed-TTS: A family of high-quality versatile speech generation models,”arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  24. [32]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,”arXiv preprint arXiv:1912.06670, 2019

    1912 arXiv

  25. [33]

    Didispeech: A large scale Man- darin speech corpus,

    T. Guo, C. Wen, D. Jiang, N. Luo, R. Zhang, S. Zhao, W. Li, C. Gong, W. Zou, K. Hanet al., “Didispeech: A large scale Man- darin speech corpus,” inProc. ICASSP, 2021, pp. 6968–6972

    2021

  26. [34]

    DNSMOS P.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “DNSMOS P.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,” inProc. ICASSP, 2022

    2022

  27. [35]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProc. ICML, 2023, pp. 28 492–28 518

    2023

  28. [36]

    Funasr: A fundamental end-to-end speech recognition toolkit,

    Z. Gao, Z. Li, J. Wang, H. Luo, X. Shi, M. Chen, Y . Li, L. Zuo, Z. Du, Z. Xiaoet al., “Funasr: A fundamental end-to-end speech recognition toolkit,”arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  29. [37]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  30. [38]

    Maskgct: Zero-shot text-to- speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text-to- speech with masked generative codec transformer,”arXiv preprint arXiv:2409.00750, 2024

    2024 arXiv

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.