Pith. sign in

REVIEW 5 major objections 7 minor 2 cited by

GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

T0 review · 5 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GenSE claims that speech enhancement is better treated as conditional language modeling over discrete tokens than as continuous signal regression, and reports state-of-the-art quality and generalization on benchmark tests.

desk verdict GenSE's real contribution is the SimCodec single-quantizer codec and the hierarchical two-stage design, but the headline quality claims rest on unexplained numeric inconsistencies and metric-only comparisons. read the letter →

arxiv 2502.02942 v1 pith:Z6YUVSHY submitted 2025-02-05 eess.AS cs.SD

Jixun Yao , Hexin Liu , Chen Chen , Yuchen Hu , EngSiong Chng , Lei Xie This is my paper · ORCID
classification eess.AScs.SD
keywords speechenhancementlanguagemodelssemantictokensacousticneuralaudiocodechierarchicalmodelingtokenchainpromptinggeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GenSE, a framework that turns speech enhancement into a conditional language-modeling task: noisy speech is first tokenized into semantic tokens and acoustic tokens, then two decoder-only language models generate clean semantic tokens and clean acoustic tokens in sequence. The authors argue that this explicitly injects semantic information—linguistic patterns and context—that conventional denoising models ignore, and that this is why GenSE outperforms discriminative and diffusion baselines on the DNS test set and generalizes better to the CHiME-4 corpus. A sympathetic reader would care because it reframes enhancement as a generation problem where the model can reconstruct masked or noisy content using language structure, rather than merely filtering a waveform. The paper reports the highest DNSMOS and speaker-similarity scores among compared systems, and the lowest word error rate on CHiME-4.

What carries the argument

The load-bearing components are discrete speech tokens and a two-stage LM hierarchy. Semantic tokens come from a pre-trained self-supervised model (XLSR) with k-means clustering (1024 clusters, 20 ms frames); acoustic tokens come from SimCodec, a single-quantizer VQ-VAE with an 8192-entry codebook obtained by training two group quantizers, sorting their embeddings by usage, concatenating the top-N and top-K entries pairwise, and fine-tuning with the reorganized codebook. The N2S language model predicts clean semantic tokens autoregressively from noisy semantic tokens, and the S2S language model predicts clean acoustic tokens from the token-chain prompt of noisy semantic, clean semantic, and noisy acoustic tokens; SimCodec's decoder then reconstructs the waveform. The hierarchy works by isolating noise removal from speech generation so the second stage sees clean content, and the token chain works by feeding speaker-specific acoustic information from the noisy input into the generation stage.

What would settle it

Run the same enhancement test on the DNS challenge set with all baselines retrained from scratch under identical data and compute budgets, then compare DNSMOS OVL with confidence intervals; if any baseline matches or exceeds GenSE's 3.43, the claim of state-of-the-art quality is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a hierarchical, token-based generative pipeline—separating denoising (noise-to-semantic, N2S) from speech generation (semantic-to-speech, S2S)—yields better enhancement quality and generalization than current state-of-the-art systems. The denoising stage is a language model that maps noisy semantic tokens to clean semantic tokens, and the generation stage is a second language model that produces clean acoustic tokens conditioned on a token chain: noisy semantic tokens, predicted clean semantic tokens, and noisy acoustic tokens, the last providing speaker timbre. On the DNS challenge test set, GenSE reports an OVL score of 3.43 without reverb and 3.19 with reverb, an SECS of 0.67 and 0.65 respectively, and on CHiME-4 a WER of 28.4, all above the baselines including the prior LM-based SELM and the diffusion models StoRM and DOSE. The paper also claims that the custom single-quantizer codec SimCodec, trained with a codebook-reorganization process, delivers competitive reconstruction at far lower token rates, which simplifies the LM prediction task.

Load-bearing premise

The paper's strongest claim depends on the non-intrusive metrics (DNSMOS, SECS, VQScore, and WER) being trustworthy proxies for perceived quality and speaker similarity of generated speech, and on the baseline results being comparable in training data and compute; if either fails, the reported superiority could shrink or disappear.

Editorial extensions

If this is right

  • If the central claim holds, speech enhancement can be formulated as two separate language-model tasks, making it compatible with the broader toolkit of decoder-only LMs, including scaling, prompting, and streaming or parallel decoding tricks.
  • A single-quantizer codec with a reorganized large codebook reduces the acoustic token sequence length, lowering the LM's prediction burden and enabling higher reconstruction quality at lower bit rates than multi-quantizer codecs.
  • Generative token-based enhancement should be more robust to domain shift than deterministic mapping models, because the prior over clean speech is learned from a distribution rather than a fixed input-output map.
  • Including noisy acoustic tokens as a prompt preserves speaker identity during generation, which the ablation links to a large jump in speaker-similarity score (SECS 0.66 with the chain versus 0.43 without it).
  • Hierarchical decoupling stabilizes LM prediction: the ablation shows removing the N2S/S2S separation drops DNSMOS OVL from 3.31 to 3.17, supporting the claim that denoising and generation should be separate stages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hierarchy and token-chain idea could transfer to other speech restoration tasks—dereverberation, bandwidth extension, or repairing clipped audio—where a noisy-condition prompt can carry the speaker or channel identity while a first stage cleans content. This is an extension the paper does not test.
  • The reported metric gap may partly reflect that the baselines' published results were not retrained under identical data and compute budgets; a matched-recipe comparison with confidence intervals would be the natural next check, and until then the superiority claim rests on comparability assumptions.
  • The token-chain prompting mechanism is essentially a conditional in-context cue; it could be studied in isolation as a general way to inject reference speaker characteristics into any speech language model, not only enhancement.
  • The autoregressive bottleneck the paper acknowledges could be addressed by alternating semantic/acoustic token prediction as the authors suggest, or by non-autoregressive acoustic generation, and the effect on quality versus speed is a testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes GenSE, a generative speech enhancement system built around language models. Speech is tokenized into XLSR-based semantic tokens and SimCodec acoustic tokens (a custom single-quantizer codec with a codebook-reorganization training scheme). A two-stage hierarchical framework first maps noisy semantic tokens to clean semantic tokens via an N2S language model, then generates clean acoustic tokens via an S2S language model conditioned on a token-chain prompt consisting of noisy semantic tokens, predicted clean semantic tokens, and noisy acoustic tokens. Experiments on the DNS test set and CHiME-4 compare GenSE with discriminative and generative baselines (FullSubNet, Inter-Subnet, CDiffuSE, SGMSE, StoRM, SELM, DOSE) using DNSMOS, SECS, VQScore, and WER, with additional subjective violin plots and ABX results; separate experiments evaluate SimCodec against several neural codecs. The paper claims state-of-the-art speech quality and generalization capability.

Significance. The architecture is timely and well motivated: treating enhancement as conditional language modeling with separated semantic-denoising and acoustic-generation stages is a plausible route to better generalization, and the single-quantizer SimCodec with reorganization addresses a practical bottleneck for LM-based generation. The paper is also unusually complete in its ablations, covering SSL extractor choice, autoregressive versus non-autoregressive N2S, codec choice, bandwidth, and quantization strategy, and the authors provide public code and demo links. If the empirical claims are substantiated, this would be a meaningful advance in LM-based speech enhancement. However, the current evidence is not yet sufficient to support the central superiority claim: the main result table is inconsistent with the analysis tables, no statistical uncertainty is reported, baseline comparability is not documented, and the chosen non-intrusive metrics have known reliability risks for generative outputs. These issues are fixable and do not indicate a fundamentally unsound method, but they must be addressed before the comparative claims can be accepted.

major comments (5)
  1. [§4.2, Tables 1 and 4–9] The GenSE row is not internally consistent across tables. Table 1 reports DNSMOS SIG/BAK/OVL of 3.65/4.18/3.43, SECS 0.67, and VQ 0.717 for the without-reverb condition, whereas Tables 4, 5, 6, 8, and 9 report the same system as 3.57/3.96/3.31 with SECS 0.66 and VQ 0.694; Table 4 additionally differs in VQ, reporting 0.649. No explanation is given for whether these are different checkpoints, different evaluation subsets (e.g., mixed reverb/no-reverb), or copy errors. Because the central claim of superiority is read from Table 1, the manuscript must reconcile these numbers and state explicitly which configuration and evaluation condition each table uses.
  2. [§4.1–4.2, Tables 1–2] No confidence intervals, standard deviations, or significance tests are reported for any of the DNSMOS, SECS, VQScore, or WER comparisons. Several margins over the strongest baselines are small enough to be within typical run-to-run or test-set variability, for example Table 1 without reverb (GenSE OVL 3.43 vs. DOSE 3.31) and Table 2 (GenSE OVL 2.89 vs. SELM 2.62 and DOSE 2.61). Please report per-utterance distributions or bootstrap confidence intervals for the main comparisons and, where possible, paired significance tests against each baseline.
  3. [Appendix A.4, Tables 1–2] The baseline descriptions are one-sentence summaries and do not establish comparability of the comparison. The paper does not state whether official released checkpoints were used, whether models were retrained on the same LibriLight/LibriTTS/VoiceBank/DNS mixtures with the same on-the-fly augmentation and SNR range, or how CHiME-4 WER was computed (which transcriptions were used, whether the same ASR checkpoint and decoding settings were applied to all systems). Without this information, the relative improvements in Tables 1 and 2 cannot be attributed to GenSE rather than to differences in training data or evaluation protocol.
  4. [§4.1 and Appendix A.1, Q2] The SE comparison relies exclusively on non-intrusive learned metrics (DNSMOS, VQScore, SECS, WER) and explicitly excludes PESQ/STOI. This is a defensible choice for generative outputs, but the paper does not establish that DNSMOS and VQScore are unbiased proxies for the perceived quality of LM-generated speech, and WER structurally favors systems that regenerate linguistically plausible tokens rather than preserving the original utterance. Please provide evidence of metric reliability for this artifact style, for example per-system correlation between objective metrics and the subjective scores in Figures 4–5, and report conventional intrusive metrics as auxiliary information for GenSE and the baselines even if they are not the primary comparison.
  5. [§4.3, Table 3] The claim that SimCodec 'significantly outperforms DAC under similar bandwidth settings (1.73 in PESQ, 0.17 in STOI, 1.09 in MCD, and 1.31 in UTMOS)' is numerically inconsistent with the table. The stated differences correspond to comparing SimCodec at 1.3 kbps with DAC at 1.0 kbps (3.05−1.32=1.73 for PESQ), not to the 0.65 kbps row that the surrounding text appears to emphasize; comparing 0.65 kbps with DAC 1.0 kbps gives 1.13 in PESQ, 0.119 in STOI, 0.92 in MCD, and 0.98 in UTMOS. Please clarify which bandwidth pair is being compared, use a common bandwidth or explicitly justify the comparison, and report significance or at least standard deviations for the codec results.
minor comments (7)
  1. [Tables 1–2] The parenthetical percentages are not defined in the captions; they appear to be relative to the FullSubNet row, and this should be stated explicitly.
  2. [Eq. (1)] Equation (1) uses p(x) on the left-hand side, but the right-hand side is a product over semantic token probabilities; please define x or rename the left-hand side to avoid confusing the speech signal with the token sequence.
  3. [Figure 4] The x-axis label 'DOES' should read 'DOSE'.
  4. [§4.1] There is a typo: 'Mel-Ceptral Distortion' should be 'Mel-Cepstral Distortion'.
  5. [Appendix A.6] The quantization-strategy comparison appears as an unnumbered 'Table' without a caption; it should be numbered and include a short description of the CVQ and FSQ baselines and their training settings.
  6. [Appendix A.3] The model configuration section does not report the reorganization counts N and K, the stage-1 codebook sizes, the SimCodec latent dimension, or the LM architecture sizes (layers, hidden dimension, heads). Please add these values or point to the released configuration files.
  7. [Figures 4–5] The subjective evaluation does not report the number of participants, the number of utterances, or the number of ratings per system; these details are needed to interpret the violin plots and ABX percentages.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: GenSE's enhancement pipeline is a learned conditional generation system evaluated against external metrics and baselines, with no derived quantity reducing to its own input.

full rationale

I walked the claimed derivation chain: XLSR semantic tokenization, N2S language-model denoising of semantic tokens, S2S acoustic-token generation conditioned on clean semantic tokens plus noisy semantic and acoustic tokens, and SimCodec decoding. None of these stages defines its output in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The token-chain prompting mechanism conditions the S2S module on noisy acoustic tokens to preserve speaker characteristics; that is an architectural design choice, not a circular derivation. The paper's comparative superiority claims are supported by external learned metrics (DNSMOS, SECS, VQScore, WER) and by baseline systems from other groups; concerns about metric bias or baseline comparability are benchmarking and validity issues, not circularity. The authors' self-citations (SELM, PromptVC, StableVC) appear as related work, baselines, or background and are not load-bearing uniqueness theorems or ansatz-smuggling citations. No equation or construction was identified in which a reported result reduces by definition to its own input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework relies on pretrained XLSR tokens being denoisable, on SimCodec's single-quantizer reconstruction being good enough, and on the evaluation metrics faithfully reflecting quality. These are domain assumptions rather than derived facts, and the paper provides only empirical evidence for them.

free parameters (4)
  • SimCodec codebook size = 8192
    Selected after the codebook-size sweep in Figure 2, trading PESQ against codebook usage; this is an experimentally tuned design parameter, not a derived quantity.
  • Acoustic token rate = 50 or 100 tokens per second (0.65 or 1.3 kbps)
    The main system uses 100 tokens/s; a 50 tokens/s variant is tested in Table 9 for speed. The choice controls the quality-efficiency tradeoff and is fit to downstream performance.
  • Reorganization selection counts N and K = not reported
    The final codebook size is N times K equals 8192, but the individual counts of top-used embeddings selected from the two stage-1 quantizers are not stated. These counts determine the quantized space and are chosen empirically.
  • SimCodec loss weights lambda1 and lambda2 = 45 and 0.1
    Set manually in Appendix A.2 for SimCodec training; no sensitivity analysis is provided.
assumptions (5)
  • domain assumption XLSR semantic tokens extracted from noisy speech retain enough phonetic and linguistic content that the N2S language model can map them to clean semantic tokens.
    Section 3.3.1 uses XLSR k-means tokens as both input and training target; the paper does not measure token corruption rates or alignment error under low-SNR noise.
  • domain assumption SimCodec's single-quantizer reconstruction quality is high enough that codec distortion does not dominate the SE error.
    Table 3 supports reconstruction quality on LibriSpeech, but the end-to-end assumption that residual codec error is negligible for SE is not separately analyzed.
  • domain assumption Noisy acoustic tokens, when used as a prompt, retain enough speaker timbre information to preserve speaker similarity after enhancement.
    Ablation in Table 4 shows a large SECS drop without them, but no analysis of how noise in acoustic tokens interacts with SNR is provided.
  • domain assumption DNSMOS, SECS, VQScore, and WER are unbiased, adequately sensitive proxies for perceived quality and speaker similarity.
    Section 4.1 defines the metrics; the paper intentionally omits PESQ and STOI for SE, and no confidence intervals or calibration against human judgments are reported.
  • domain assumption Baseline systems are evaluated under comparable training data, augmentation, and compute conditions.
    Section 4.1 lists baselines and Appendix A.4 describes them only at a high level; the paper does not state whether baseline numbers were reproduced in GenSE's training pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling." pith.science (2026). https://pith.science/paper/Z6YUVSHY

@misc{pith2026250202942,
  author       = {Pith},
  title        = {Pith review of: GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6YUVSHY}},
  note         = {Machine review of arXiv:2502.02942}
}
read the original abstract

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called \textit{GenSE}. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.

Figures

Figures reproduced from arXiv: 2502.02942 by the authors.

Figure 1
Figure 1. The hierarchical modeling framework of language model in GenSE. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The relationship between reconstruction quality and codebook usage across different code￾book sizes. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The detailed architecture and training process of SimCodec, with the reorganization pro [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Violin plots for speech naturalness and speaker similarity, comparing the signals enhanced [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: ABX results between GenSE and baseline systems. “No Preference” means that partici [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Comparison between SimCodec and state-of-the-art acoustic codec model. The vertical [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the enhanced spectrograms generated by different models for a severely [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    UniPASE extends the PASE framework with DeWavLM-Omni to convert degraded speech into high-fidelity, low-hallucination audio across sampling rates via phonetic enhancement, acoustic adaptation, and multi-rate vocoding.

  2. GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

    eess.AS 2025-12 conditional novelty 5.0 of 10

    A two-stage decoder-only language model with continuous embeddings and UTMOS-based preference fine-tuning reports improved target-speaker-extraction scores on Libri2Mix.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding

    Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling. Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402.10533,

    arXiv

  2. [2]

    While increasing the number of quantizers improves enhancement performance when using either autoregressive and non-autoregressive hybrid prediction or parallel prediction, the results still do not surpass the performance achieved by employing SimCodec. Furthermore, we compare the performance between GenSE with current bandwidth and lower band- width and ...

    work page 2025

  3. [4]

    Unsupervised cross-lingual representation learning for speech recognition

    Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979,

    arXiv 2006

  4. [5]

    High fidelity neural audio compression

    Alexandre D ´efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438,

    arXiv

  5. [6]

    Polyvoice: Language models for speech to speech transla- tion

    Qianqian Dong, Zhiying Huang, Qiao Tian, Chen Xu, Tom Ko, Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, et al. Polyvoice: Language models for speech to speech transla- tion. arXiv preprint arXiv:2306.02982,

    arXiv

  6. [7]

    The spectrogram produced by GenSE shows clear harmonic structures and speech formants, demonstrating its ability to effec- tively suppress noise and retain speech details. In contrast, the variant models without hierarchical modeling exhibit more distortion and residual noise, which indicates inferior noise reduction and signal reconstruction compared to ...

    work page 2025

  7. [8]

    Variational autoencoder for speech enhancement with a noise-aware encoder

    Huajian Fang, Guillaume Carbajal, Stefan Wermter, and Timo Gerkmann. Variational autoencoder for speech enhancement with a noise-aware encoder. In ICASSP 2021-2021 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pp. 676–680,

    work page 2021

  8. [11]

    Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement

    Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6633–6637,

    work page 2021

Show all 48 references
  1. [12]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    11 Published as a conference paper at ICLR 2025 Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transaction...

    2025

  2. [13]

    Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement

    Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement. arXiv preprint arXiv:2212.11377,

    arXiv

  3. [14]

    Language-codec: Reducing the gaps between discrete codec representation and speech language models

    Shengpeng Ji, Minghui Fang, Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. Language-codec: Reducing the gaps between discrete codec representation and speech language models. arXiv preprint arXiv:2402.12208, 2024a. Shengpeng Ji, Ziyue Jiang, Xize Cheng, Y...

    arXiv

  4. [16]

    Libri- light: A benchmark for asr with limited or no supervision

    Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazar´e, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna...

    2020

  5. [17]

    A study on data augmentation of reverberant speech for robust speech recognition

    Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE interna- tional conference on acoustics, speech and signal processing (ICASSP) , pp. 5220–5224,

    2017

  6. [19]

    Lan- guage models as controlled natural language semantic parsers for knowledge graph question an- swering

    Jens Lehmann, Preetam Gattogi, Dhananjay Bhandiwad, S ´ebastien Ferr´e, and Sahar Vahdati. Lan- guage models as controlled natural language semantic parsers for knowledge graph question an- swering. In ECAI 2023, pp. 1348–1356

    2023

  7. [20]

    Single-codec: Single-codebook speech codec towards high-performance speech generation

    Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. Single-codec: Single-codebook speech codec towards high-performance speech generation. arXiv preprint arXiv:2406.07422,

    arXiv

  8. [21]

    Noise tokens: Learning neural noise templates for environment- aware speech enhancement

    Haoyu Li and Junichi Yamagishi. Noise tokens: Learning neural noise templates for environment- aware speech enhancement. arXiv preprint arXiv:2004.04001,

    2004 arXiv

  9. [23]

    V oicefixer: Toward general speech restoration with neural vocoder

    Haohe Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yux- uan Wang. V oicefixer: Toward general speech restoration with neural vocoder. arXiv preprint arXiv:2109.13731,

    arXiv

  10. [24]

    Se- manticodec: An ultra low bitrate semantic audio codec for general sound

    Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Se- manticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233, 2024a. Hexin Liu, Xiangyu Zhang, Leibny Paola Garcia, Andy WH Khong, Eng Siong Chng, and Sh...

    2022 arXiv

  11. [25]

    Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement

    Soumi Maiti and Michael I Mandel. Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement. In ICASSP 2020-2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP) , pp. 206–210,

    2020

  12. [26]

    Finite scalar quantiza- tion: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023a. Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made s...

    arXiv

  13. [28]

    Dnsmos p

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 886–890,

    2022

  14. [29]

    Fewer-token neural speech codec with time-invariant codes

    Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12737–12741. IEEE,

    2024

  15. [30]

    arXiv preprint arXiv:2204.02152,

    arXiv

  16. [31]

    Universal score-based speech enhancement with high content preservation

    Robin Scheibler, Yusuke Fujita, Yuma Shirahata, and Tatsuya Komatsu. Universal score-based speech enhancement with high content preservation. arXiv preprint arXiv:2406.12194,

    arXiv

  17. [32]

    V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis

    13 Published as a conference paper at ICLR 2025 Hubert Siuzdak. V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814,

    2025 arXiv

  18. [33]

    The voice bank corpus: Design, collection and data analysis of a large regional accent speech database

    Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research an...

    2013

  19. [35]

    Wham!: Extending speech separation to noisy environments

    Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint arXiv:1907.01160,

    1907 arXiv

  20. [36]

    Audiodec: An open-source streaming high-fidelity neural audio codec

    Yi-Chiao Wu, Israel D Gebru, Dejan Markovi´c, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1–5,

    2023

  21. [37]

    Hifi-codec: Group-residual vector quantization for high fidelity audio codec

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765, 2023a. Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xua...

    arXiv

  22. [38]

    Libritts: A corpus derived from librispeech for text-to-speech

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882,

    1904 arXiv

  23. [39]

    Speaking in wavelet domain: A simple and efficient approach to speed up speech diffusion model

    Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia, Eng Siong Chng, and Lina Yao. Speaking in wavelet domain: A simple and efficient approach to speed up speech diffusion model. arXiv preprint arXiv:2402.10642,

    arXiv

  24. [40]

    Speechtokenizer: Unified speech tokenizer for speech large language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692,

    arXiv

  25. [41]

    We discuss several common questions for the design of GenSE

    A A PPENDIX In the supplemental material: • A.1. We discuss several common questions for the design of GenSE. • A.2. We provide training objectives of SimCodec. • A.3. We provide the model configurations of SimCodec and LM. • A.4. We describe the details of baseline systems em...

    2022

  26. [42]

    On the other hand, we chose XLSR as the semantic extractor due to the key advantage of its multilin- gual speech representation capabilities

    We can find that GenSE achieves similar results when using these two different SSL models. On the other hand, we chose XLSR as the semantic extractor due to the key advantage of its multilin- gual speech representation capabilities. While other SSL models also offer robustness...

    2025

  27. [43]

    To address this limitation, we also conducted an ABX test to assess the perceptual quality of the enhanced speech compared to clean speech, as shown in Figure

    and (Hsu et al., 2022). To address this limitation, we also conducted an ABX test to assess the perceptual quality of the enhanced speech compared to clean speech, as shown in Figure

    2022

  28. [44]

    No Preference

    This evaluation provides a more subjective measure of quality, capturing how close the generated speech sounds to clean speech. Question 3 : How does the performance of GenSE compare with clean samples from a subjective perceived aspect? We conduct an ABX test to compare the p...

    2025

  29. [47]

    We also observe that finite scalar quantization (FSQ) (Mentzer et al., 2023a) demonstrates lower reconstruction quality

    in PESQ, STOI, and MCD metrics, with only a slight degradation in UTMOS. We also observe that finite scalar quantization (FSQ) (Mentzer et al., 2023a) demonstrates lower reconstruction quality. We attribute this to several factors: the smaller latent dimension of the vector in...

    2024

  30. [128]

    We employ the AdamW optimizer with a learning rate of 1e-4 to optimize the codec model

    The SimCodec model is trained for 50k steps in the first stage and 10k steps in the second stage. We employ the AdamW optimizer with a learning rate of 1e-4 to optimize the codec model. For language model training, we use 8 A100 GPUs with a batch size of 256, training for 1 mi...

    2025

  31. [2013]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023a. Xiaofei Wang, Manthan Thakker, Zhuo C...

    arXiv

  32. [2015]

    Interspeech 2021 deep noise suppression challenge

    Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. Interspeech 2021 deep noise suppression challenge. arXiv preprint arXiv:2101.01902,

    2021 arXiv

  33. [2016]

    Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec

    Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 591–595,

    2024

  34. [2017]

    Nu-gan: High resolution neural upsampling with gan

    Rithesh Kumar, Kundan Kumar, Vicki Anand, Yoshua Bengio, and Aaron Courville. Nu-gan: High resolution neural upsampling with gan. arXiv preprint arXiv:2010.11362,

    2010 arXiv

  35. [2019]

    Self-supervised speech qual- ity estimation and enhancement using only clean speech.arXiv preprint arXiv:2402.16321,

    Szu-Wei Fu, Kuo-Hsuan Hung, Yu Tsao, and Yu-Chiang Frank Wang. Self-supervised speech qual- ity estimation and enhancement using only clean speech.arXiv preprint arXiv:2402.16321,

    arXiv

  36. [2020]

    Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthe- sis

    12 Published as a conference paper at ICLR 2025 Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthe- sis. arXiv preprint arXiv:2411.01156,

    2025 arXiv

  37. [2021]

    Metricgan: Generative adversarial net- works based black-box metric scores optimization for speech enhancement

    Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. Metricgan: Generative adversarial net- works based black-box metric scores optimization for speech enhancement. In International Conference on Machine Learning, pp. 2031–2041. PmLR,

  38. [2022]

    Audiolm: a language modeling approach to audio generation

    Zal´an Borsos, Rapha¨el Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, ...

    2023 arXiv

  39. [2023]

    Naturalspeech 3: Zero-shot speech synthesis with factor- ized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factor- ized codec and diffusion models. arXiv preprint arXiv:2403.03100,

    arXiv

  40. [2024]

    Unsupervised speech en- hancement using dynamical variational autoencoders.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2993–3007,

    10 Published as a conference paper at ICLR 2025 Xiaoyu Bie, Simon Leglaive, Xavier Alameda-Pineda, and Laurent Girin. Unsupervised speech en- hancement using dynamical variational autoencoders.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2993–3007,

    2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.