Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Query-Based Asymmetric Modeling with Decoupled Input-Output Rates for Speech Restoration

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One asymmetric model restores speech across decoupled input-output sampling rates without resampling.

desk verdict A genuinely useful architecture and a solid empirical study, but the 'arbitrary rates' claim overreaches and the PESQ/UTMOS training-evaluation overlap needs attention. read the letter →

arxiv 2509.21003 v4 pith:5MCJUXXN submitted 2025-09-25 eess.AS

Ui-Hyeop Shin , Jaehyun Ko , Woocheol Jeong , Hyung-Min Park This is my paper · ORCID
classification eess.AS
keywords speechrestorationsampling-frequency-independentbandwidthextensionasymmetricencoder-decoderqueriestime-frequencytransformeradversarialtrainingstreamingenhancement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that speech restoration does not need to be tied to a single input-output sampling rate. It proposes TF-Restormer, a model that encodes only the observed frequency band of a degraded signal and then synthesizes the missing high-frequency band through learned extension queries, so one trained model can denoise, dereverberate, extend bandwidth, and undo combined distortions at multiple input and output rates without resampling. A shared sampling-frequency-independent STFT discriminator and a scaled log-spectral loss allow the model to be trained adversarially across rates while preserving fine spectral detail. If the claim holds, a single non-diffusion model can replace a collection of rate-specific restoration systems, and a streaming variant can do the same in real time.

What carries the argument

The load-bearing object is the decoupled SFI-STFT formulation, in which a fixed 40 ms window with 20 ms hop makes the number of frequency bins scale with the sampling rate, so input and output rates obey $f_E:f_D = (F_E-1):(F_D-1)$. On top of this sits the asymmetric encoder-decoder: a time-frequency dual-path encoder analyzes only the $F_E$ input bins, while a lightweight decoder pads the frequency axis to $F_D$ with learnable extension queries shared across frames and refines them with a frequency cross-self module whose cross-attention uses encoder features as key and value, a band-partitioned cross-attention mechanism. The same frequency projection layer is shared across frequency modules to inject a spectral-structure inductive bias, and the SFI-STFT discriminator applies adversarial supervision with STFT windows of fixed physical duration across all rates. This combination is what lets one model synthesize missing high-frequency content at user-chosen output rates without resampling.

What would settle it

Evaluate the trained model on input-output rate pairs absent from training, for example 8→22.05 kHz or 32→48 kHz, and compare LSD, MCD, and UTMOS against the same model's results on trained rate pairs; a clear quality collapse on unseen rate ratios would refute the arbitrary-rate claim.

Watch

Extended reading notes

Core claim

TF-Restormer's central claim is that a single asymmetric encoder-decoder operating on a sampling-frequency-independent short-time Fourier transform can restore speech at decoupled input-output rates, with no external resampling and no systematic loss of fidelity or perceptual quality relative to rate-specific systems. The encoder performs heavy time-frequency analysis only on the observed input band, while the decoder reconstructs the unobserved high-frequency band from learnable extension queries whose cross-attention keys and values come from the encoder; because the STFT uses a constant frame duration, frequency-bin counts scale with the sampling rates, making the internal representation rate-agnostic. Trained with a perceptual loss, a scaled log-spectral loss, and a shared SFI-STFT discriminator, the same model handles denoising, dereverberation, bandwidth extension, and combined distortions, and its causal time module yields a streaming variant. The reported evaluations show it matching or beating dedicated per-rate baselines on these tasks while keeping compute proportional to the input band rather than the output rate.

Load-bearing premise

The load-bearing premise is that extension queries trained only on 8/16 kHz inputs and 16/24/44.1/48 kHz outputs will still synthesize correctly at any other user-chosen rate, because the SFI formulation alone does not guarantee that unseen frequency-bin counts and alignments are handled well.

Editorial extensions

If this is right

  • A single TF-Restormer model covers the trained rate pairs (8/16 kHz inputs to 16/24/44.1/48 kHz outputs), so a deployment can drop per-rate models and the associated resampling pipeline.
  • Compute scales with the input band: 8→16 kHz uses roughly half the MACs of 16→16 kHz, because heavy analysis is never applied to bins the input does not contain.
  • The streaming variant, built by replacing the time module with a causal Mamba block, keeps near-offline quality with roughly 80 ms latency, making real-time universal restoration feasible.
  • Combined distortions such as noise, reverberation, clipping, codecs, and band limitation can be repaired in one pass at the chosen output rate, instead of chaining a denoiser and a separate bandwidth-extension model.
  • The shared SFI-STFT discriminator allows adversarial training to be unified across rates, so adding a new supported output rate does not require training a new discriminator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'arbitrary rate' claim is only as strong as the interpolation: the model is trained with output rates {16,24,44.1,48} kHz and inputs {8,16} kHz, so rates such as 22.05 or 32 kHz are an inference. A test on unseen rates would settle whether the ratio identity and shared extension queries generalize.
  • The same architecture could be pointed at arbitrary spectral holes, not just a high-frequency band: because extension queries are learnable and cross-attend to the observed band, one could mask arbitrary frequency regions and let the decoder inpaint them, turning the model into a general spectral inpainter.
  • The asymmetric encoder-decoder suggests a separable capacity budget: an application that needs better high-frequency detail for a fixed input rate could enlarge the decoder (or its number of extension queries) without recomputing the encoder, which the ablations hint at by showing decoder design matters mainly when rates differ.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TF-Restormer, a single encoder-decoder model for speech restoration under decoupled input and output sampling rates (which the authors call the xSFI setting). The encoder uses a time-frequency dual-path Transformer to analyze the input bandwidth, while a lightweight decoder synthesizes the missing high-frequency band using learnable extension queries and cross-attention over encoder features. A shared sampling-frequency-independent STFT discriminator and a scaled log-spectral loss are introduced to support multi-rate training and perceptual quality. The model is trained on VCTK with input rates of 8 and 16 kHz and output rates of 16, 24, 44.1, and 48 kHz, and evaluated on UNIVERSE, VCTK-DEMAND, a VCTK super-resolution set, and real-recorded VoxCeleb data, including a streaming variant. The paper claims consistent improvements over prior, frequently larger baselines with lower computational cost for low input rates, and reports ablations for the loss, decoder design, and frequency projection.

Significance. If the empirical claims hold, TF-Restormer is a valuable contribution to speech restoration: a single non-diffusion model that handles denoising, dereverberation, bandwidth extension, and combined distortions across multiple sampling rates, with a streaming mode and openly available code and demos. The architectural asymmetry of a heavy encoder and a lightweight decoder with extension queries is well motivated, and the shared SFI-STFT discriminator is a sensible way to train a single generator across rates. The scaled log-spectral loss is clearly explained and ablated. The main reservations concern the breadth of the 'arbitrary rates' claim, the overlap between the training objective and the headline evaluation metrics, and the absence of uncertainty estimates in the reported tables.

major comments (4)
  1. [Sec. 3.2, Sec. 4, Table 3] The claim of 'arbitrary input-output sampling rates' is not supported. The extension query q_ext in Sec. 3.2 is a learnable parameter tensor with shape (F_D−F_E)×T×C_D, so its size is tied to the specific rate pair; no mechanism such as interpolation, conditioning, or generation is described for obtaining queries for extension lengths not seen in training. Training uses only f_E in {8,16} kHz and f_D in {16,24,44.1,48} kHz (Sec. 4), and Table 3 evaluates only these combinations; the rates 22.05 and 32 kHz, which Sec. 3.1 lists as supported, are never tested. Since an 8 to 32 kHz extension would require 480 extension bins and training only sees extension lengths of 160, 320, 562, 640, 722, and 800, generalization to arbitrary user-specified rates is nontrivial and remains unverified. Additionally, the case f_D < f_E is undefined because q_ext would have negative dimension. Please either scope the claim to the trained rate pairs or add a rate-generalization mechanism and evaluate it on unseen rates such as 22.05 and 32 kHz.
  2. [Eq. (6), Tables 1-3] The training objective in Eq. (6) includes L_hf = L_pesq + 10·L_utmos, while Tables 1-3 report PESQ and UTMOS as primary outcomes. Consequently, a non-trivial part of the reported advantage on these specific metrics is directly optimized during training, which is a form of evaluation leakage. The paper should disclose this overlap explicitly in the main text of Section 5 (it is currently not stated there) and provide results with the human-feedback loss ablated, or justify why the PESQ and UTMOS gains are not artifacts of this term. The independent metrics (WVMOS, DNSMOS, LSD, MCD, sBERT) mitigate the concern but do not eliminate it, because the headline tables foreground PESQ and UTMOS.
  3. [Tables 1-3] No error bars, confidence intervals, or significance tests are reported for any of the main quantitative results. Several comparisons hinge on small differences, for example Table 3 clean 8 to 16 kHz LSD is 0.89 for TF-Restormer versus 0.83 for NVSR, while the UTMOS advantage of 4.10 versus 3.64 is only one column; it is therefore difficult to assess whether 'consistently outperforms' is robust. Please add variability estimates across utterances or random seeds, or at least report the number of test utterances per cell and pairwise significance tests for the central claims.
  4. [Sec. 5.1, Appendix D, Table 3] The super-resolution test set is generated by a simulation pipeline that follows the same procedure used for training, which the paper acknowledges 'may provide a slight advantage to our model.' Since this is the only evaluation supporting the core multi-rate capability, the advantage is not necessarily slight: the model is tested on the distribution on which it was trained, while the baselines may be at a disadvantage if their training distributions differ. Please either include a cross-dataset or held-out evaluation, such as a public bandwidth-extension test set or native 44.1/48 kHz recordings, or temper the universality claims accordingly.
minor comments (5)
  1. [Sec. 3.2] The definition of q_ext as R^{(F_D−F_E)×T×C_D} conflicts with the statement that its values are 'shared across all the frames'; if the same query is shared, the tensor should be R^{(F_D−F_E)×1×C_D} and tiled, or the text should clarify the intended sharing mechanism.
  2. [Sec. 3.3] The shape notation is inconsistent: earlier the STFT features are R^{F×T×2}, but Sec. 3.3 writes 'feature with shape of R^{T×F×C}'. Since the modules treat time and frequency differently, please standardize the axis order.
  3. [Table 1] FINALLY is listed with '—' for PESQ, SDR, LSD, MCD, sBERT, and sTokDis; this incomplete comparison makes it impossible to verify the 'balanced gains' claim against the strongest perceptual baseline. Please add the missing metrics or indicate unavailability in the table.
  4. [Table 8] The entries for FINALLY contain typographical artifacts ('4.05±±±0.07', '3.98± ±±0.06', '3.31± ±±0.04'); please correct these.
  5. [Figure 5] The x-axis label reads 'distrance d' instead of 'distance d', and the unit ×10^{-3} appears inconsistent with the text examples of w; please fix the label and clarify the units.

Circularity Check

1 steps flagged · score 6.0 of 10

PESQ and UTMOS are used both in the training loss and as headline evaluation metrics, so part of the reported advantage is optimized directly rather than predicted; the architectural derivation itself is otherwise self-contained.

  1. fitted input called prediction [Section 4.2, Eq. (6); reported in Tables 1–3]
    "Lgen(θ) = λ_g L_g(θ) + λ_fm L_fm(θ) + λ_p L_p(θ) + λ_s L_s(θ) + λ_hf L_hf(θ), (6) ... where L_hf = L_pesq + 10·L_utmos is additional human-feedback loss (Babaev et al., 2024) for aesthetic quality with differentiable PESQ loss and UTMOS loss (Saeki et al., 2022)."

    The tables used to support the paper's central claim report PESQ and UTMOS as key perceptual-fidelity outcomes, but these are exactly the two quantities composing L_hf in the generator's adversarial objective. The model is therefore gradient-trained against the same functions that are later presented as evidence of superiority, so the PESQ/UTMOS gains are partly by construction rather than emergent prediction. This is the fitted-input-called-prediction pattern. The circularity is only partial: λ_hf is small (0.0001), and the paper also reports independent metrics not present in the loss, including LSD, MCD, sBERT, sTokDis, WVMOS, and DNSMOS, plus ablations that do not rely on this term.

full rationale

TF-Restormer's core architecture—SFI-STFT front end, encoder analysis, extension-query decoder, and MHCA conditioning—is derived from stated components and is not defined in terms of its reported outcomes; the f_E:f_D = (F_E-1):(F_D-1) relation is just constant-frame-duration STFT mathematics. No uniqueness theorem or load-bearing self-citation chain is invoked: the only in-house citation (Shin et al., 2025) appears as a related-work example, not as support for the central claim. The one genuine circular element is Eq. (6), where L_hf adds differentiable PESQ and UTMOS losses to the generator objective while Tables 1–3 use PESQ and UTMOS as headline metrics. That makes those particular reported gains partly optimized directly rather than independently predicted. The circularity is bounded: λ_hf = 0.0001, and multiple independent metrics and ablations provide outside grounding. The paper's 'arbitrary input-output rates' wording is broader than the trained/evaluated rate set, and extension queries are length-specific learnable parameters with no described mechanism for unseen rate combinations; that is an overclaim or evidence gap, but it is not a circular reduction, so it does not raise the circularity score further.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on a set of architectural choices and empirical assumptions; no physical constants or mathematical axioms are needed. The main free parameters are training-loss weights and the empirical scale of the proposed loss; the main unproved assumptions are SFI bin alignment, rate generalization, surrogate-metric validity, and simulation fidelity.

free parameters (4)
  • Scaled log-spectral component weights alpha_m, alpha_r, alpha_i = 0.6, 0.2, 0.2
    Hand-set in Eq. (4) to weight magnitude versus real and imaginary terms in the proposed loss; the loss is central to reported gains in Table 4a.
  • Per-frame scale w_tf for scaled log-spectral loss = E[S_m,tf], the frame-averaged source magnitude
    Chosen empirically in Section 4.1 as the expected magnitude of the target; it controls where gradients are suppressed in Eq. (4).
  • Loss weights lambda_p, lambda_s, lambda_g, lambda_fm, lambda_hf = 100, 1, 0.005, 0.1, 0.0001
    Selected by hand in Section 4; these balance the objectives and directly determine the optimized solution, so they are free parameters relative to the performance claims.
  • Frequency projection dimension F_proj = 512
    Chosen for Eq. (1) and (2) in the attention-with-structural-bias module; a hand-set architectural hyperparameter.
assumptions (4)
  • domain assumption SFI-STFT frame consistency: a fixed 40 ms window and 20 ms hop yields valid STFT/iSTFT transformations for all rates whose bin counts satisfy f_E:f_D = (F_E-1):(F_D-1).
    Invoked in Section 3.1 to define xSFI; assumes spectral bins of different rates align physically and that padding and extension queries can bridge bin-count differences without resampling.
  • ad hoc to paper Rate generalization: behavior learned on training rates {8,16} kHz input and {16,24,44.1,48} kHz output transfers to arbitrary user-specified rates.
    Section 4 samples only these rates, yet Section 3.1 and the Abstract promise arbitrary input-output rates; no experiments outside the training rate set are provided.
  • domain assumption Surrogate metric validity: WavLM features and differentiable PESQ and UTMOS surrogates are adequate proxies for human perceptual quality.
    Used in Eq. (3) and Eq. (6); the field relies on such proxies, but using the same proxies as losses and evaluation metrics introduces shared bias.
  • domain assumption Simulation fidelity: the RIR, noise, codec, clipping, and filtering pipeline in Appendix B approximates real-world recording conditions.
    The model is trained and, for VCTK super-resolution, evaluated on simulations generated by the same procedure, so the results depend on that simulation being representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Query-Based Asymmetric Modeling with Decoupled Input-Output Rates for Speech Restoration." pith.science (2026). https://pith.science/paper/5MCJUXXN

@misc{pith2026250921003,
  author       = {Pith},
  title        = {Pith review of: Query-Based Asymmetric Modeling with Decoupled Input-Output Rates for Speech Restoration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MCJUXXN}},
  note         = {Machine review of arXiv:2509.21003}
}
read the original abstract

Speech restoration aims to recover clean speech from degraded recordings affected by noise, reverberation, bandwidth reduction, or other distortions, where input and output sampling rates may differ. Existing approaches typically assume matched input-output rates and apply redundant resampling, limiting native multi-rate processing. We formulate this gap as the extended sampling-frequency-independent (xSFI) setting, where a model must operate under decoupled input-output rates, and propose TF-Restormer, a query-based xSFI modeling framework. The model encodes only the observed input band and synthesizes the unobserved high-frequency band through extension queries with band-partitioned cross-attention, yielding an asymmetric encoder-decoder that allocates capacity to analysis while keeping synthesis lightweight. Trained with a perceptual loss, a scaled log-spectral loss, and adversarial supervision via an SFI-STFT discriminator, TF-Restormer attains balanced fidelity-perceptual quality as a single unified model, without redundant resampling across denoising, dereverberation, bandwidth extension, and combined distortion benchmarks under multiple sampling rates.

Figures

Figures reproduced from arXiv: 2509.21003 by the authors.

Figure 1
Figure 1. Overview of TF-Restormer. The model adopts an encoder-decoder design: TF-Encoder performs input-bandwidth analysis through stacked time and frequency modules, while the TF-Decoder reconstructs missing high-frequency bands with lightweight processing and learnable extension queries. 2025), invariably begin by resampling the input to the target rate before processing. This design choice simplifies model training but i… view at source ↗
Figure 2
Figure 2. Overall architecture of TF-Restormer. With SFI-STFT and iSTFT, arbitrary input-output sample rates can be addressed in a single model. The encoder feature serves as both the input and the fixed key/value for the Freq. cross-self module in the TF-Decoder. 3.2 ANALYSIS ENCODER AND EXTENSION DECODER TF-encoder for input analysis As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Unit modules in TF-Encoder and TF-Decoder. The (a) time module is based on MHSA with RoPE while (b) the frequency encoder module is based on MHSA with frequency projection layer. (c) The frequency decoder module employs MHCA based on key/value from the encoder features Attention with structural bias Linformer (Wang et al., 2020) introduced linear projections of key-value to reduce the computations of attention, whil… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Noisy-distorted speech input simulation pipeline. The simulation procedure is partitioned to phys￾ical distortion and digital distortion. Then, three kinds of digital distortions were simulated in sequence. We randomly applied audio clipping, crystalizer, flanger, and …
Figure 5
Figure 5. Figure 5: Gradient profiles of the proposed scaled log-spectral loss [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Unit modules in TF-Encoder and TF-Decoder. The (a) time module is based on MHSA with RoPE while (b) the frequency encoder module is based on MHSA with frequency projection layer. (c) The frequency decoder module utilize MHCA based on key/value from the encoder features…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 33 canonical work pages

  1. [1]

    Hifi++: a unified framework for bandwidth extension and speech enhancement

    Pavel Andreev, Aibek Alanov, Oleg Ivanov, and Dmitry Vetrov. Hifi++: a unified framework for bandwidth extension and speech enhancement. arXiv preprint arXiv:2203.13086, 2022

    arXiv 2022

  2. [2]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016

    2016

  3. [3]

    FINALLY : fast and universal speech enhancement with studio-like quality

    Nicholas Babaev, Kirill Tamogashev, Azat Saginbaev, Ivan Shchekotov, Hanbin Bae, Hosang Sung, WonJun Lee, Hoon-Young Cho, and Pavel Andreev. FINALLY : fast and universal speech enhancement with studio-like quality. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=18RdkSv9h9

    work page 2024

  4. [4]

    CMGAN: Conformer-based Metric GAN for Speech Enhancement

    Ruizhe Cao, Sherif Abdulatif, and Bin Yang. CMGAN: Conformer-based Metric GAN for Speech Enhancement . In Proc. Interspeech, pp.\ 936--940, 2022. doi:10.21437/Interspeech.2022-517

    doi:10.21437/interspeech.2022-517 2022

  5. [5]

    An investigation of incorporating mamba for speech enhancement

    Rong Chao, Wen-Huang Cheng, Moreno La Quatra, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Szu-Wei Fu, and Yu Tsao. An investigation of incorporating mamba for speech enhancement. arXiv preprint arXiv:2405.06573, 2024

    arXiv 2024

  6. [6]

    Wav LM : L arge-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wav LM : L arge-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 0 (6): 0 1505--1518, 2022

    work page 2022

  7. [7]

    Phase-aware speech enhancement with deep complex u-net

    Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee. Phase-aware speech enhancement with deep complex u-net . In Proc. Int. Conf. Learn. Represent. (ICLR), 2018

    work page 2018

  8. [9]

    Music source separation in the waveform domain

    Alexandre D \'e fossez, Nicolas Usunier, L \'e on Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254, 2019

    arXiv 1911

Show all 72 references
  1. [10]

    High fidelity neural audio compression

    Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=ivCd8z8zR2. Featured Certification, Reproducibility Certification

    2023

  2. [11]

    Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator

    Yariv Ephraim and David Malah. Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator. IEEE Transactions on acoustics, speech, and signal processing, 32 0 (6): 0 1109--1121, 1984

    1984

  3. [12]

    Fukada, K

    T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai. An adaptive algorithm for mel-cepstral analysis of speech. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp.\ 137--140 vol.1, 1992. doi:10.1109/ICASSP.1992.225953

    1992

  4. [13]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2

    2024

  5. [14]

    Siamese masked autoencoders

    Agrim Gupta, Jiajun Wu, Jia Deng, and Li Fei-Fei. Siamese masked autoencoders. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=yC3q7vInux

    2023

  6. [15]

    Woods, Ivo Merks, and Tao Zhang

    Kun Han, Yuxuan Wang, DeLiang Wang, William S. Woods, Ivo Merks, and Tao Zhang. Learning spectral mapping for speech dereverberation and denoising. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23 0 (6): 0 982--992, 2015. doi:10.1109/TASLP.2015.2416653

    2015

  7. [16]

    Nu-wave 2: A general neural audio upsampling model for various sampling rates

    Seungu Han and Junhyeok Lee. Nu-wave 2: A general neural audio upsampling model for various sampling rates. In INTERSPEECH, pp.\ 4401--4405, 2022. URL https://doi.org/10.21437/Interspeech.2022-45

    2022 doi

  8. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll\'ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 16000--16009, June 2022

    2022

  9. [18]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33: 0 6840--6851, 2020

    2020

  10. [19]

    DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement

    Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement . In Proc. Interspeech, pp.\ 2472--2476, 2020. doi:10.21437/Interspeech.2020-2537

    2020 doi

  11. [20]

    A binaural room impulse response database for the evaluation of dereverberation algorithms

    Marco Jeub, Magnus Schafer, and Peter Vary. A binaural room impulse response database for the evaluation of dereverberation algorithms. In 2009 16th International Conference on Digital Signal Processing, pp.\ 1--5, 2009. doi:10.1109/ICDSP.2009.5201259

    2009

  12. [21]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiangyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and sheng zhao. Naturalspeech 3: Zero-shot speech synthesis with fa...

    2024

  13. [22]

    Istftnet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform

    Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. Istftnet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp...

    2022

  14. [23]

    Audio super-resolution with robust speech representation learning of masked autoencoder

    Seung-Bin Kim, Sang-Hoon Lee, Ha-Yeong Choi, and Seong-Whan Lee. Audio super-resolution with robust speech representation learning of masked autoencoder. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 0 1012--1022, 2024. doi:10.1109/TASLP.2023.3349053

    2024

  15. [24]

    Miipher: A robust speech restoration model integrating self-supervised speech and text representations

    Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Yu Zhang, Wei Han, Ankur Bapna, and Michiel Bacchiani. Miipher: A robust speech restoration model integrating self-supervised speech and text representations. In 2023 IEEE Workshop on Applicat...

    2023

  16. [25]

    Hifi-GAN : Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-GAN : Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33: 0 17022--17033, 2020 a

    2020

  17. [26]

    Diffwave: A versatile diffusion model for audio synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020 b

    2009 arXiv

  18. [27]

    Mel GAN : Generative adversarial networks for conditional waveform synthesis

    Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. Mel GAN : Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing s...

    2019

  19. [28]

    Reducing the prior mismatch of stochastic differential equations for diffusion-based speech enhancement

    Bunlong Lay, Simon Welker, Julius Richter, and Timo Gerkmann. Reducing the prior mismatch of stochastic differential equations for diffusion-based speech enhancement. arXiv preprint arXiv:2302.14748, 2023

    2023 arXiv

  20. [29]

    Nu-wave: A diffusion probabilistic model for neural audio upsampling

    Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio upsampling. In Interspeech 2021, pp.\ 1634--1638, 2021. doi:10.21437/Interspeech.2021-36

    2021 doi

  21. [30]

    Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation

    Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  22. [31]

    MaskSR: Masked Language Model for Full-band Speech Restoration

    Xu Li, Qirui Wang, and Xiaoyu Liu. MaskSR: Masked Language Model for Full-band Speech Restoration . In Interspeech 2024 , pp.\ 2275--2279, 2024. doi:10.21437/Interspeech.2024-1584

    2024 doi

  23. [32]

    Neural vocoder is all you need for speech super-resolution

    Haohe Liu, Woosung Choi, Xubo Liu, Qiuqiang Kong, Qiao Tian, and DeLiang Wang. Neural vocoder is all you need for speech super-resolution. In Interspeech 2022, pp.\ 4227--4231, 2022 a . doi:10.21437/Interspeech.2022-11017

    2022 doi

  24. [33]

    Voicefixer: A unified framework for high-fidelity speech restoration

    Haohe Liu, Xubo Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: A unified framework for high-fidelity speech restoration. In Interspeech 2022, pp.\ 4232--4236, 2022 b . doi:10.21437/Interspeech.2022-11026

    2022 doi

  25. [34]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization . In Proc. Int. Conf. Learn. Represent. (ICLR), 2019

    2019

  26. [35]

    MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra

    Ye-Xin Lu, Yang Ai, and Zhen-Hua Ling. MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra . In Proc. Interspeech, pp.\ 3834--3838, 2023

    2023

  27. [36]

    Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction

    Ye-Xin Lu, Yang Ai, Hui-Peng Du, and Zhen-Hua Ling. Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction. IEEE Transactions on Audio, Speech and Language Processing, 33: 0 236--250, 2025. doi:10.1109/TASLP.2024.3519881

    2025

  28. [37]

    Understanding and improving transformer from a multi-particle dynamic system point of view

    Yiping Lu*, Zhuohan Li*, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019....

    2020

  29. [38]

    Wolfgang Mack and Emanuël A. P. Habets. Declipping speech using deep filtering. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp.\ 200--204, 2019. doi:10.1109/WASPAA.2019.8937287

    2019

  30. [39]

    Least squares generative adversarial networks

    Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.\ 2794--2802, 2017

    2017

  31. [40]

    VoxCeleb : a large-scale speaker identification dataset

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb : a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017

    2017 arXiv

  32. [41]

    Acoustical sound database in real environments for sound scene understanding and hands-free speech recognition

    Satoshi Nakamura, Kazuo Hiyane, Futoshi Asano, Takanobu Nishiura, and Takeshi Yamada. Acoustical sound database in real environments for sound scene understanding and hands-free speech recognition. In M. Gavrilidou, G. Carayannis, S. Markantonatou, S. Piperidis, and G. Stainha...

    2000

  33. [42]

    WaveNet : A generative model for raw audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet : A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016

    2016 arXiv

  34. [43]

    SEGAN : Speech enhancement generative adversarial network

    Santiago Pascual, Antonio Bonafonte, and Joan Serrà. SEGAN : Speech enhancement generative adversarial network. In Interspeech 2017, pp.\ 3642--3646, 2017. doi:10.21437/Interspeech.2017-1428

    2017 doi

  35. [44]

    Sampling frequency independent dialogue separation

    Jouni Paulus and Matteo Torcoli. Sampling frequency independent dialogue separation. In 2022 30th European Signal Processing Conference (EUSIPCO), pp.\ 160--164, 2022. doi:10.23919/EUSIPCO55093.2022.9909824

    2022

  36. [45]

    Chandan K.A. Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, Puneet Rana, Sriram Srinivasan, and Johannes Gehrke. The INTERSPEECH 2020 Deep Noise Suppression Challenge: Data...

    2020 doi

  37. [46]

    Dnsmos p

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 886--890. IEEE, 2022

    2022

  38. [47]

    Speech enhancement and dereverberation with diffusion-based generative models

    Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 0 2351--2364, 2023. doi:10.1109/TASLP.2023.3285241

    2023

  39. [48]

    Rix, J.G

    A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra. Perceptual evaluation of speech quality ( PESQ )-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings...

    2001

  40. [49]

    Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. SDR - half-baked or well done? CoRR, abs/1811.02508, 2018. URL http://arxiv.org/abs/1811.02508

    2018 arXiv

  41. [50]

    UTMOS : Utokyo-sarulab system for voicemos challenge 2022

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS : Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022

    2022 arXiv

  42. [51]

    SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics

    Takaaki Saeki, Soumi Maiti, Shinnosuke Takamichi, Shinji Watanabe, and Hiroshi Saruwatari. SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics . In Interspeech 2024 , pp.\ 4943--4947, 2024. doi:10.21437/Interspeech.2024-1508

    2024 doi

  43. [52]

    Germain, Zexu Pan, and Jonathan Le Roux

    Kohei Saijo, Gordon Wichern, François G. Germain, Zexu Pan, and Jonathan Le Roux. Tf-locoformer: Transformer with local modeling by convolution for speech separation and enhancement. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC), pp.\ 205--209, 20...

    2024

  44. [53]

    Universal score-based speech enhancement with high content preservation

    Robin Scheibler, Yusuke Fujita, Yuma Shirahata, and Tatsuya Komatsu. Universal score-based speech enhancement with high content preservation. In Interspeech 2024, pp.\ 1165--1169, 2024. doi:10.21437/Interspeech.2024-138

    2024 doi

  45. [54]

    Universal speech enhancement with score-based diffusion

    Joan Serr \`a , Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.03065, 2022

    2022 arXiv

  46. [55]

    TF-CorrNet : Leveraging spatial correlation for continuous speech separation

    Ui-Hyeop Shin, Bon Hyeok Ku, and Hyung-Min Park. TF-CorrNet : Leveraging spatial correlation for continuous speech separation. IEEE Signal Processing Letters, 32: 0 1875--1879, 2025. doi:10.1109/LSP.2025.3562819

    2025

  47. [56]

    Wave-u-net: A multi-scale neural network for end-to-end audio source separation

    Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. CoRR, abs/1806.03185, 2018. URL http://arxiv.org/abs/1806.03185

    2018 arXiv

  48. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. ISSN 0925-2312. doi:https://doi.org/10.1016/j.neucom.2023.127063. URL https://www.sciencedirect.com/sc...

    2024

  49. [58]

    HiFi-GAN : High-fidelity denoising and dereverberation based on speech deep features in adversarial networks

    Jiaqi Su, Zeyu Jin, and Adam Finkelstein. HiFi-GAN : High-fidelity denoising and dereverberation based on speech deep features in adversarial networks. 2020

    2020

  50. [59]

    HiFi-GAN-2 : Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features

    Jiaqi Su, Zeyu Jin, and Adam Finkelstein. HiFi-GAN-2 : Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features. In 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp.\ 166--170. IEEE, 2021

    2021

  51. [60]

    The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings

    Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. Proceedings of Meetings on Acoustics, 19 0 (1): 0 035081, 05 2013. ISSN 1939-800X. doi:10.1...

    2013 doi

  52. [61]

    MLP -mixer: An all- MLP architecture for vision

    Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP -mixer: An all- MLP architecture for vision. In A. Beygelzimer, Y....

    2021

  53. [62]

    Noisy speech database for training speech enhancement algorithms and tts models

    Cassia Valentini-Botinhao et al. Noisy speech database for training speech enhancement algorithms and tts models. 2017

    2017

  54. [63]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020

    2020

  55. [64]

    Deep learning based target cancellation for speech dereverberation

    Zhong-Qiu Wang and DeLiang Wang. Deep learning based target cancellation for speech dereverberation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 0 941--950, 2020. doi:10.1109/TASLP.2020.2975902

    2020

  56. [65]

    TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation

    Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe. TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation . IEEE/ACM Trans. Audio, Speech, Language Process., 31: 0 3221--3236, 2023. doi:10.1109/TASLP.2023.3304482

    2023

  57. [66]

    Speech enhancement with score-based generative models in the complex STFT domain

    Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based generative models in the complex STFT domain. In Proc. Interspeech 2022, pp.\ 2928--2932, 2022. doi:10.21437/Interspeech.2022-10653

    2022 doi

  58. [67]

    Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92)

    Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019

    2019

  59. [68]

    Dual-branch attention-in-attention transformer for single-channel speech enhancement

    Guochen Yu, Andong Li, Yutian Wang, Yinuo Guo, Hui Wang, and Chengshi Zheng. Dual-branch attention-in-attention transformer for single-channel speech enhancement. CoRR, abs/2110.06467, 2021. URL https://arxiv.org/abs/2110.06467

    2021 arXiv

  60. [69]

    Toward Universal Speech Enhancement For Diverse Input Conditions

    Wangyou Zhang, Kohei Saijo, Zhong-Qiu Wang, Shinji Watanabe, and Yanmin Qian. Toward Universal Speech Enhancement For Diverse Input Conditions . In Proc. IEEE ASRU, pp.\ 1--6, 2023. doi:10.1109/ASRU57964.2023.10389733

    2023

  61. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  62. [71]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  63. [72]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  64. [73]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.