Pith. sign in

REVIEW 3 major objections 5 minor 46 references

AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single masked-autoencoder Transformer can analyze, control, and generate speech, mapping a Mel-spectrogram to six editable attributes and back, and demonstrates this on pitch estimation, pitch shifting, and…

desk verdict A solid consolidation: one bidirectional masked autoencoder for speech analysis, control, and generation, with credible experiments and some fixable evaluation gaps; the pitch-shift metric is teacher-consistency, but the f0 robustness result is independently validated. read the letter →

arxiv 2501.05332 v1 pith:Z2TPCB6D submitted 2025-01-09 cs.SD cs.AIeess.AS

Samir Sadok , Simon Leglaive , Laurent Girin , Gaël Richard , Xavier Alameda-Pineda This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords maskedautoencoderspeechanalysisandsynthesisattributespitchestimationshiftingenhancementdiscretetokensMel-spectrogram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that one masked-autoencoder Transformer can do what previously required separate analysis and synthesis systems: it maps a Mel-spectrogram into six speech attributes—linguistic content, pitch, loudness, speaker identity, signal-to-noise ratio, and clarity index—and maps those attributes back into a Mel-spectrogram. Because the same model learns both directions, editing an attribute before resynthesis yields controllable speech transformation. The paper demonstrates this on pitch estimation, pitch shifting, and denoising, reporting accuracy that rivals specialized systems while keeping a single set of weights. If the central claim holds, a single trained model could replace pipelines that chain an analyzer, a controller, and a vocoder.

What carries the argument

The load-bearing object is a single encoder–decoder Transformer that auto-encodes the concatenation of two discrete token streams: a VQ-VAE-tokenized Mel-spectrogram and tokenized sequences for each of the six attributes. The scalar attributes are normalized, resampled, rounded, and grouped into discrete tokens, while the HuBERT content embedding and the speaker embedding are quantized with $k$-means. A coupled masking strategy randomly splits the masking budget between the two streams during training, followed by an all-or-nothing phase in which one stream is fully masked; this teaches both inter- and intra-representation dependencies and makes the same model work in both directions. A pre-trained HiFi-GAN vocoder converts the generated Mel-spectrograms into speech waveforms.

What would settle it

Train AnCoGen on speech where CREPE's fundamental-frequency estimates are systematically biased, such as under severe reverberation or with an added offset, then compare AnCoGen's estimated pitch to laryngograph ground truth on the PTDB-TUG corpus; if the error follows CREPE's bias instead of ground truth, the robust-estimation claim would be overturned.

Watch

Extended reading notes

Core claim

The central claim is that a masked autoencoder can learn a bidirectional mapping between two tokenized representations of the same utterance: the Mel-spectrogram and a set of six speech attributes. During training, tokens from both representations are randomly masked—sometimes partially in both, sometimes one entire side—so the Transformer learns to reconstruct missing tokens from whichever side is visible. At inference, entirely masking the attributes yields analysis, entirely masking the Mel-spectrogram yields generation, and modifying selected attribute tokens between the two steps yields control. The paper reports that this one model estimates fundamental frequency accurately in noise and reverberation, shifts pitch with accuracy competitive with WORLD and TD-PSOLA, and denoises speech while preserving quality better than dedicated enhancement networks, at the cost of weaker speaker-identity preservation for unseen speakers.

Load-bearing premise

The training labels for pitch, content, and speaker identity come from pretrained models—CREPE, HuBERT, and ECAPA-TDNN—so AnCoGen's analysis quality is bounded by their accuracy, and the pitch-modification evaluation uses CREPE's estimates as the target rather than ground-truth pitch.

Editorial extensions

If this is right

  • A single AnCoGen model can replace separate analysis and synthesis components for the covered tasks, since the same weights produce attributes from audio and audio from attributes.
  • Pitch estimation is a byproduct of analysis and remains accurate down to 0 dB SNR with reverberation, with average absolute error below 6.7 Hz in all tested conditions, unlike classic trackers pYin and SWIPE.
  • Pitch shifting by +10%, -10%, +50%, and -50% keeps average absolute error below about 6 Hz against CREPE's estimates while preserving perceived quality.
  • Setting the SNR attribute to 40 dB suppresses noise with stable quality under mismatched noise conditions, where dedicated denoisers degrade more.
  • Unseen speakers are only approximately reproduced because speaker identity is quantized to the 251 training speakers, which the paper states as the main limitation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: swapping the speaker-identity attribute tokens before resynthesis should provide voice conversion, and the paper's companion website gestures at this; a controlled listening test with unseen speakers would test whether output speaker similarity scales with codebook size.
  • Editorial inference: the same bidirectional masking recipe could be applied to other paired representations, such as text or articulatory features, effectively turning the masked autoencoder into a generic controllable codec.
  • Editorial inference: because the training attributes come from pretrained teachers, the model's analysis ceiling is set by those teachers; notably, the pitch-modification evaluation measures agreement with CREPE rather than laryngograph ground truth.
  • Editorial inference: denoising by clamping the SNR attribute occasionally changes phonetic content, so applications demanding word-for-word fidelity would need an intelligibility constraint or a content-preserving loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AnCoGen, a masked-autoencoder-based Transformer that learns a bidirectional mapping between a Mel-spectrogram and six quantized speech attributes (HuBERT content tokens, CREPE pitch, RMS loudness, ECAPA-TDNN speaker identity, Brouhaha SNR, and Brouhaha C50). At inference, fully masking one representation lets the same model perform analysis (Mel-spectrogram to attributes) and generation (attributes to Mel-spectrogram), followed by a HiFi-GAN vocoder. The authors evaluate the method on analysis-resynthesis, robust f0 estimation, pitch shifting, and speech denoising, comparing against WORLD/TD-PSOLA for pitch control and several neural enhancement systems for denoising.

Significance. If the central claims hold, AnCoGen is a conceptually attractive simplification: one Transformer with frozen tokenizers and a vocoder can analyze, control, and generate speech. The paper has several genuine strengths: code and audio examples are available; the robust f0 estimation experiment uses laryngograph ground truth on PTDB-TUG, providing an independent check; the denoising evaluation includes matched and mismatched conditions with several reference systems; and the architecture is described in enough detail to reimplement. The main quantitative weakness is that the pitch-modification experiment (Table II) is evaluated with CREPE, which is also the teacher that generated the pitch training labels, so the headline pitch-control result is partly a teacher-consistency check rather than an independent validation. This is addressable with additional evaluation, but it is load-bearing for the claim of precise pitch control.

major comments (3)
  1. [III-B.3, Table II] The pitch-shifting experiment evaluates AAE between the desired and predicted f0 using CREPE, while Section II-A states that CREPE also generated the training labels for attribute A2. AnCoGen is therefore being scored on how well it reproduces its teacher's estimate, not on whether the resynthesized waveform has the target f0 as a physical property. A model that memorized or closely fit CREPE's input-output mapping could achieve low AAE on this metric while failing an independent pitch measurement. Please add an evaluation against laryngograph ground truth, as already done in the robust f0 estimation experiment (Section III-B.2), or against another independent pitch estimator, and report any differences; this is necessary to support the claim of precise pitch modification.
  2. [III-B.1, Table I] The analysis-resynthesis experiment compares AnCoGen only with feeding the ground-truth Mel-spectrogram directly into HiFi-GAN. This comparison quantifies the loss introduced by the discrete tokenization and the attribute bottleneck, but it does not situate AnCoGen against prior attribute-based analysis/resynthesis systems, such as a pipeline using the same off-the-shelf attribute estimators with a separate synthesis model. Adding such a baseline, or an existing speech-resynthesis method, would make the claim that AnCoGen provides high-quality analysis and resynthesis in a unified model more convincing.
  3. [II-C] The masking strategy is described as having two consecutive phases (coupled masking followed by all-or-nothing masking), but the paper does not specify the relative duration of each phase or how the transition is scheduled. Since the ability to perform both analysis and generation with one model depends on this schedule, please state the number of epochs or fraction of training devoted to each phase, or cite the specific schedule used in the multimodal MAE reference [16] if it is adopted verbatim.
minor comments (5)
  1. [II-B] For the scalar attributes A2, A3, A5, and A6, the paper gives the grouping widths Di and codebook sizes Ki but not the quantization ranges or bin boundaries. Please specify the normalization and rounding procedure so that the tokenization is fully reproducible.
  2. [References] References [16] and [22] are the same paper (a vector quantized masked autoencoder for audiovisual speech emotion recognition) and should be consolidated into a single entry.
  3. [III-B.4, Table III] AnCoGen ranks last on COS for speaker identity preservation. The discussion cites [46] to justify prioritizing overall quality, but it would be informative to also report speaker similarity with a second embedding model (e.g., a different verification network) to confirm that the low COS is not specific to Resemblyzer.
  4. [III-A] The training details give an epoch time of 4 minutes on 4 A100 GPUs but do not state the total number of epochs used for each task or whether all tasks use the same 800-epoch budget. Clarify this in the setup.
  5. [III-B.2, Fig. 3] Please specify whether the AAE values are computed over voiced frames only or over all frames, since the choice of unvoiced-frame handling can substantially affect pitch-tracking metrics.

Circularity Check

1 steps flagged · score 4.0 of 10

Pitch-shift evaluation uses CREPE both as training label source and as test-time estimator, making the AAE a teacher-consistency measure; robust-f0 and quality metrics remain independent.

  1. fitted input called prediction [Section II-A (speech representations) and Section III-B (Pitch shifting)]
    "A2: The pitch contour f0 (in Hz), as estimated by the pre-trained CREPE model [18] ... we evaluate the performance in terms of AAE between the desired and predicted f0 (using CREPE)."

    The f0 attribute A2 is produced for training by CREPE, so AnCoGen's analysis path is a supervised distillation of CREPE's estimate. In the pitch-shifting experiment, the 'desired' f0 is a shifted version of CREPE's estimate of the input, and the 'predicted' f0 is CREPE's estimate of the resynthesized signal. Both sides of the reported AAE are therefore generated by the same pretrained estimator that supplied the training labels. The metric measures how consistently AnCoGen can reproduce its teacher through the quantized attribute codebook, Mel-spectrogram reconstruction, and HiFi-GAN vocoder, rather than whether the output waveform has the target f0 as an independently measured physical property.

full rationale

The central architecture is self-contained as a bidirectional masked-autoencoder mapping between Mel-spectrograms and discrete attribute tokens, and most evaluations use independent references: robust f0 estimation is scored against laryngograph ground truth, resynthesis quality uses N-MOS/WER/COS, and denoising uses standard intrusive and non-intrusive metrics. The only substantively circular step is the pitch-shifting evaluation, where CREPE supplies the training labels for A2 and also computes both the desired and predicted f0 values in the AAE, so the pitch-modification number partly reduces to teacher consistency rather than an independent measurement. The self-citations to the authors' prior VQ-MAE work are methodological and not load-bearing for the main empirical claims. Overall, the paper has partial, task-specific circularity in one headline result, but the central claim is supported by independent experiments elsewhere.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The model builds on pretrained attribute estimators as ground truth, which introduces a ceiling on analysis quality and a degree of circularity.

free parameters (4)
  • K1 = 100 k-means clusters for HuBERT content = 100
    Chosen by hand (Section II-B) to quantize the content representation.
  • Grouping widths Di for attribute tokens = (2,4,4,4,4,4)
    Chosen to balance token length; no systematic search reported.
  • Masking ratio distribution (uniform on 1-simplex) = uniform p
    Training strategy from multimodal MAE; no ablation justifying the uniform choice.
  • Target SNR of 40 dB for denoising = 40 dB
    Set as the desired clean-speech SNR attribute during enhancement (Section III-B).
assumptions (4)
  • domain assumption HuBERT embeddings represent linguistic content
    Section II-A adopts this without justification.
  • domain assumption CREPE f0 estimates are a valid training target for pitch
    Section II-A uses CREPE to label f0; if CREPE is biased, the model learns the bias.
  • domain assumption The VQ-VAE codebook preserves enough spectral detail for synthesis
    The Mel-spectrogram is tokenized by a pretrained frozen VQ-VAE; the paper assumes no critical information loss.
  • ad hoc to paper Speaker identity can be quantized to 251 clusters
    Section II-B sets K4 to the number of training speakers, which causes the recognized unseen-speaker limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder." pith.science (2026). https://pith.science/paper/Z2TPCB6D

@misc{pith2026250105332,
  author       = {Pith},
  title        = {Pith review of: AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2TPCB6D}},
  note         = {Machine review of arXiv:2501.05332}
}
read the original abstract

This article introduces AnCoGen, a novel method that leverages a masked autoencoder to unify the analysis, control, and generation of speech signals within a single model. AnCoGen can analyze speech by estimating key attributes, such as speaker identity, pitch, content, loudness, signal-to-noise ratio, and clarity index. In addition, it can generate speech from these attributes and allow precise control of the synthesized speech by modifying them. Extensive experiments demonstrated the effectiveness of AnCoGen across speech analysis-resynthesis, pitch estimation, pitch modification, and speech enhancement.

Figures

Figures reproduced from arXiv: 2501.05332 by the authors.

Figure 1
Figure 1. Analysis, control, and generation of speech with AnCoGen. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of AnCoGen, to read from left (input) to right (output). [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. f0 estimation results. is completed with a learned mask token before being sent to the decoder. The decoder output is a “full” sequence of decoded/predicted discrete tokens that can be used for MS and SAs estimation (using the VQ-VAE decoder and inverse quantization, respectively). C. Masking strategy Training AnCoGen is achieved by randomly masking the input sequences of MS and SA tokens. The masking ratio (i.e., t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 36 canonical work pages

  1. [14]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv:2104.00355, 2021

    arXiv 2021

  2. [1]

    J. D. Markel and A. J. Gray, Linear prediction of speech . New York: Springer-Verlag, 1976

    work page 1976

  3. [2]

    Speech analysis/synthesis based on a sinusoidal representation,

    R. McAulay and T. Quatieri, “Speech analysis/synthesis based on a sinusoidal representation,” IEEE Trans. Audio, Speech, Lang. Proc. , vol. 34, no. 4, pp. 744–754, 1986

    work page 1986

  4. [3]

    Speech analysis/synthesis and modifica- tion using an analysis-by-synthesis/overlap-add sinusoidal model,

    E. B. George and M. J. Smith, “Speech analysis/synthesis and modifica- tion using an analysis-by-synthesis/overlap-add sinusoidal model,” IEEE Trans. Audio, Speech, Lang. Proc. , vol. 5, no. 5, pp. 389–406, 1997

    work page 1997

  5. [4]

    Spectral modeling synthesis: A sound analy- sis/synthesis system based on a deterministic plus stochastic decomposi- tion,

    X. Serra and J. Smith, “Spectral modeling synthesis: A sound analy- sis/synthesis system based on a deterministic plus stochastic decomposi- tion,” Computer Music J. , vol. 14, no. 4, pp. 12–24, 1990

    work page 1990

  6. [5]

    HNS: Speech modification based on a harmonic + noise model,

    J. Laroche, Y . Stylianou, and E. Moulines, “HNS: Speech modification based on a harmonic + noise model,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 1993

    work page 1993

  7. [6]

    Analysis/synthesis and modification of the speech aperiodic component,

    G. Richard and C. d’Alessandro, “Analysis/synthesis and modification of the speech aperiodic component,” Speech Com. , vol. 19, no. 3, pp. 221–244, 1996

    work page 1996

  8. [7]

    Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,

    E. Moulines and F. Charpentier, “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,” Speech Com. , vol. 9, no. 5-6, pp. 453–467, 1990

    work page 1990

Show all 46 references
  1. [8]

    Improved phase vocoder time-scale modi- fication of audio,

    J. Laroche and M. Dolson, “Improved phase vocoder time-scale modi- fication of audio,” IEEE Trans. Speech Audio Proc. , vol. 7, no. 3, pp. 323–332, 1999

    1999

  2. [9]

    STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually isomorphic decomposition of speech sounds,

    H. Kawahara, “STRAIGHT, exploitation of the other aspect of VOCODER: Perceptually isomorphic decomposition of speech sounds,” Acoust. Science Technol. , vol. 27, no. 6, pp. 349–353, 2006

    2006

  3. [10]

    World: a vocoder-based high- quality speech synthesis system for real-time applications,

    M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based high- quality speech synthesis system for real-time applications,” IEICE Trans. Inform. Systems , vol. 99, no. 7, pp. 1877–1884, 2016

    2016

  4. [11]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Trans. Audio, Speech, Lang. Proc. , vol. 30, pp. 495–507, 2021

    2021

  5. [12]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res. , 2023

    2023

  6. [13]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech, Lang. Proc., vol. 29, pp. 3451–3460, 2021

    2021

  7. [15]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in IEEE/CVF Conf. Computer Vision Pattern Recog. (CVPR) , 2022

    2022

  8. [17]

    Hifi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Adv. Neural Inform. Proc. Syst. (NeurIPS) , 2020

    2020

  9. [18]

    CREPE: A convolutional representation for pitch estimation,

    J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “CREPE: A convolutional representation for pitch estimation,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2018

    2018

  10. [19]

    Ecapa-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” arXiv preprint arXiv:2005.07143 , 2020

    2005 arXiv

  11. [20]

    Brouhaha: Multi- task training for voice activity detection, speech-to-noise ratio, and c50 room acoustics estimation,

    M. Lavechin, M. M ´etais, H. Titeux, A. Boissonnet, J. Copet, M. Rivi `ere, E. Bergelson, A. Cristia, E. Dupoux, and H. Bredin, “Brouhaha: Multi- task training for voice activity detection, speech-to-noise ratio, and c50 room acoustics estimation,” in IEEE Auto. Speech Recog. ...

    2023

  12. [21]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” in Adv. Neural Inform. Proc. Syst. (NeurIPS) , 2017

    2017

  13. [22]

    A vector quantized masked autoencoder for audiovisual speech emotion recognition,

    S. Sadok, S. Leglaive, and R. S ´eguier, “A vector quantized masked autoencoder for audiovisual speech emotion recognition,” arXiv preprint arXiv:2305.03568, 2023

    2023 arXiv

  14. [23]

    A comparison of discrete and soft speech units for improved voice conversion,

    B. van Niekerk, M.-A. Carbonneau, J. Za ¨ıdi, M. Baas, H. Seut ´e, and H. Kamper, “A comparison of discrete and soft speech units for improved voice conversion,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP), 2022

    2022

  15. [24]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” in Int. Conf. Learn. Repres. (ICLR) , 2020

    2020

  16. [25]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural Inform. Proc. Syst. (NeurIPS) , 2017

    2017

  17. [26]

    DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement,

    Y . Hu, Y . Liu, S. Lv, M. Xing, S. Zhang, Y . Fu, J. Wu, B. Zhang, and L. Xie, “DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement,” arXiv:2008.00264, 2020

    2008 arXiv

  18. [27]

    Phase- aware speech enhancement with deep complex U-net,

    H.-S. Choi, J.-H. Kim, J. Huh, A. Kim, J.-W. Ha, and K. Lee, “Phase- aware speech enhancement with deep complex U-net,” in Int. Conf. Learn. Repres. (ICLR), 2018

    2018

  19. [28]

    Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation,

    J. Chen, Q. Mao, and D. Liu, “Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation,” arXiv preprint arXiv:2007.13975 , 2020

    2007 arXiv

  20. [29]

    Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Proc. , vol. 27, no. 8, pp. 1256–1266, 2019

    2019

  21. [30]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2015

    2015

  22. [31]

    DEMAND: a collection of multi- channel recordings of acoustic noise in diverse environments,

    J. Thiemann, N. Ito, and E. Vincent, “DEMAND: a collection of multi- channel recordings of acoustic noise in diverse environments,” in Int. Congr . Acoust. (ICA), 2013

    2013

  23. [32]

    Pedalboard,

    P. Sobot, “Pedalboard,” jul 2021, 10.5281/zenodo.7817838. [Online]. Available: https://doi.org/10.5281/zenodo.7817838

    2021 doi

  24. [33]

    Librimix: An open-source dataset for generalizable speech separation,

    J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, “Librimix: An open-source dataset for generalizable speech separation,” arXiv preprint arXiv:2005.11262 , 2020

    2005 arXiv

  25. [34]

    Wham!: Extending speech separation to noisy environments,

    G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux, “Wham!: Extending speech separation to noisy environments,” in Interspeech, 2019

    2019

  26. [35]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  27. [36]

    SDR–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR–half-baked or well done?” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2019

    2019

  28. [37]

    Perceptual evaluation of speech quality (PESQ) – a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ) – a new method for speech quality assessment of telephone networks and codecs,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2001

    2001

  29. [38]

    An algorithm for intelligibility prediction of time–frequency weighted noisy speech,

    C. Taal, R. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” IEEE Trans. Audio, Speech, Lang. Proc. , vol. 19, no. 7, pp. 2125–2136, 2011

    2011

  30. [39]

    Generalized end-to-end loss for speaker verification,

    L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP), 2018

    2018

  31. [40]

    Speech quality assessment through MOS using non-matching references,

    P. Manocha and A. Kumar, “Speech quality assessment through MOS using non-matching references,” in Interspeech, 2022

    2022

  32. [41]

    Torchaudio-Squim: Reference-less speech quality and intelligibility measures in Torchaudio,

    A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu, “Torchaudio-Squim: Reference-less speech quality and intelligibility measures in Torchaudio,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP), 2023

    2023

  33. [42]

    DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. Reddy, V . Gopal, and R. Cutler, “DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2022

    2022

  34. [43]

    A pitch tracking corpus with evaluation on multipitch tracking scenario,

    G. Pirker, M. Wohlmayr, S. Petrik, and F. Pernkopf, “A pitch tracking corpus with evaluation on multipitch tracking scenario,” in Interspeech, 2011

    2011

  35. [44]

    pYIN: A fundamental frequency estimator using probabilistic threshold distributions,

    M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimator using probabilistic threshold distributions,” in IEEE Int. Conf. Acoust., Speech, Sig. Proc. (ICASSP) , 2014

    2014

  36. [45]

    A sawtooth waveform inspired pitch estimator for speech and music,

    A. Camacho and J. G. Harris, “A sawtooth waveform inspired pitch estimator for speech and music,” J. Acoust. Soc. Am. , vol. 124, no. 3, pp. 1638–1652, 2008

    2008

  37. [46]

    Revise: Self- supervised speech resynthesis with visual input for universal and generalized speech regeneration,

    W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y . Adi, “Revise: Self- supervised speech resynthesis with visual input for universal and generalized speech regeneration,” in IEEE/CVF Conf. Computer Vision Pattern Recog. (CVPR) , 2023

    2023

  38. [47]

    Generating diverse high- fidelity images with VQ-V AE-2,

    A. Razavi, A. Van den Oord, and O. Vinyals, “Generating diverse high- fidelity images with VQ-V AE-2,” in Adv. Neural Inform. Proc. Syst. (NeurIPS), 2019

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.