Pith. sign in

REVIEW 4 major objections 6 minor 46 references

Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Whilter claims a single Whisper-based utterance classifier can flag multi-speaker audio, non-English speech, background music, noise, and synthetic speech in one pass, replacing several single-purpose data filters for speech corpora.

desk verdict A useful multitask filter plus a solid new AITW annotation set, but the abstract overstates the F1 claim and the lack of inter-annotator agreement makes the synthetic-speech numbers provisional. read the letter →

arxiv 2507.21642 v2 pith:BZIX3N5G submitted 2025-07-29 cs.SD cs.LGeess.AS

William Ravenscroft , George Close , Kit Bower-Morris , Jamie Stacey , Dmitry Sityaev , Kris Y. Hong This is my paper
classification cs.SDcs.LGeess.AS
keywords datafilteringmulti-taskclassificationWhisperencoderin-the-wildspeechcorporatext-to-speechcleaningsyntheticdetectionattentionpoolingAITWdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single utterance-level classifier, Whilter, can replace the usual stack of separate filters needed to clean large in-the-wild speech corpora before training text-to-speech or speech-recognition models. Whilter is built on the frozen encoder of Whisper, a large speech-recognition foundation model, with a small learned transformer and five attention-pooling heads, one per undesirable property: multiple speakers, non-target languages, background music, noisy speech, and synthetic speech. To train and test it, the authors annotated roughly 21,000 clips from the Emilia and YODAS corpora and published this AITW dataset, after an initial training stage on artificially mixed audio. On their test set the strongest reported results are F1 scores of 86.7% and 87.7% for multispeaker and foreign-language detection, with equal error rates between 6.5% and 7.8% on three subtasks, and Whilter outperforms a fine-tuned BEATs audio foundation model on the three speech-specific classes at a processing time of about 0.033 seconds per clip. The paper concludes that a speech-specific foundation model is the right shared substrate for these five filtering problems.

What carries the argument

The load-bearing object is the Whilter network itself: a frozen Whisper-small encoder with learned weights over its twelve layer outputs, followed by a four-layer, four-head transformer that maps 768-dimensional frames to 256 dimensions, and a bank of five linear attention-pooling heads, each producing a binary score. The attention-pooling head, a temporal attention weighting followed by a projection with a mean-residual connection, is what lets each subtask attend to the frames relevant to it. A two-stage training schedule carries the argument: first, dynamic mixing of clean speech, music, noise, foreign-language, and synthetic-speech corpora at signal-to-noise ratios randomly chosen in the range from minus 5 to 10 dB, so every class occurs often enough to learn; then, fine-tuning on the human-labelled AITW subset of Emilia and YODAS, with weighted random sampling to offset sparse labels.

What would settle it

Re-annotate a random sample of the AITW test clips with a larger panel, for example five or more annotators, using the same guidelines and take the majority vote; if Whilter's per-class F1 against the original two-annotator labels is substantially higher than its F1 against the panel majority, especially for synthetic speech and noise, then the reported numbers largely measure label noise rather than class presence. A complementary check is to construct test audio with known ground truth by mixing clean speech with music and noise at controlled signal-to-noise ratios and adding clips from known text-to-speech systems, then verifying that Whilter's detections agree with the construction labels on the classes whose real-world labels are uncertain.

Watch

Extended reading notes

Core claim

The paper's central claim is that five speech-corpora filtering subtasks, multi-speaker detection, foreign-language detection, background-music detection, noise detection, and synthetic-speech detection, can be solved jointly by one utterance-level model: a frozen Whisper encoder whose twelve intermediate layers are combined with learned weights, a four-layer transformer prediction network, and one linear attention-pooling head per class. Trained first on dynamically mixed non-in-the-wild audio and then fine-tuned on the new AITW annotations, Whilter reaches F1 of 86.7% for multi-speaker detection and 87.7% for foreign-language detection, with equal error rates of 6.7%, 6.5%, and 7.8% on those two classes plus music. It outperforms the fine-tuned BEATs baseline on the three speech-specific subtasks, two of which BEATs cannot solve at all, and does so far faster than a chain of single-task models. The authors present this as the first all-in-one filter for TTS-oriented data cleaning and as evidence that speech-specific foundation features are preferable to general audio features for speech-defined classes.

Load-bearing premise

The load-bearing premise is that the two annotators' labels in the AITW dataset are trustworthy ground truth for all five classes, especially the sparse synthetic-speech and ambiguous noise labels; no inter-annotator agreement is reported, and the paper itself notes these labels are hard to assign with confidence.

Editorial extensions

If this is right

  • Filtering a large untranscribed corpus for English TTS becomes a single forward pass of one roughly 91-million-parameter model rather than running diarization, language identification, anti-spoofing, and music/noise detectors separately.
  • Because each of the five heads outputs a separate score, downstream tasks can apply different thresholds or keep/drop rules, for example keeping multi-speaker data for ASR while discarding non-English and synthetic speech for TTS, without retraining.
  • Speech-specific pre-trained features appear to be necessary for language and synthetic-speech detection: the BEATs baseline, even with Whilter's prediction head, collapses to chance on those two tasks, while Whilter does not.
  • The published AITW labels provide a common benchmark for future in-the-wild data filters, allowing direct comparison of multitask and single-task cleaning approaches.
  • The reported processing time of about 0.033 seconds per clip on an A10 GPU makes corpus-scale filtering practical, on the order of tens of clips per second per GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The model's usefulness for ASR data cleaning is a natural extension the paper only gestures at: the same labels, especially multi-speaker and foreign language, are exactly the failure modes that produce misaligned transcripts in self-training pipelines, so Whilter could be dropped into ASR curation with minimal changes.
  • Because the synthetic-speech head is trained on sparse, low-confidence labels, a bootstrap loop where Whilter proposes positive candidates for human review and is retrained on the confirmed set is a concrete way to test whether model-assisted annotation can escape the label scarcity the paper identifies.
  • The paper compares Whilter only to task-specific baselines and to one general audio foundation model; replacing only the frozen encoder while keeping the identical transformer-plus-attention heads would isolate how much of the gain comes from Whisper features versus the multitask architecture itself.
  • Music and noise were the two classes where BEATs beat Whilter, suggesting a hybrid system that routes general audio classes to an audio foundation model and speech classes to Whilter could outperform either alone, a direction the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents Whilter, a multi-task classifier built from a frozen Whisper encoder, a learned transformer prediction network, and attention-pooling heads that simultaneously detect five undesirable properties in in-the-wild speech: multiple speakers, foreign-language speech, background music, noisy speech, and synthetic speech. The authors also release AITW, a 64-hour annotated subset of the Emilia and YODAS corpora, together with a Label Studio interface, and they compare Whilter against several single-task and multi-task baselines. Whilter is reported to outperform a fine-tuned BEATs model on the three speech-specific subtasks, to match or exceed the evaluated single-task models on most metrics, and to offer reduced average processing time relative to single-task pipelines.

Significance. If the results hold, Whilter would be a practical one-model filter for cleaning TTS/ASR training data from in-the-wild corpora, and the released AITW dataset would be a useful community resource for further research. The paper's strengths include its comparison against multiple baselines, including a BEATs variant equipped with the same prediction head for fairness, the use of a frozen Whisper encoder with learnable layer weights, dynamic mixing for simulated training data, and a clear focus on a real-world engineering problem. However, the quantitative claims are partially overstated in the abstract, and the reliability of the ground-truth labels for synthetic speech and noise is not established, which tempers confidence in the claimed advantage over BEATs on those classes.

major comments (4)
  1. [Abstract] The abstract states that Whilter 'achieves F1 scores above 85% and equal error rates of 6.5% to 7.8% for three of five subtasks.' Table 1 shows that only the multispeaker (F1=86.7) and foreign-language (F1=87.7) subtasks have F1 above 85%; music is 77.9, noise 63.2, and synthetic speech 45.6. The stated EER range applies to three subtasks, but the sentence as written ties the F1 claim to those same three, which is inaccurate. Please revise the abstract and related text to report per-class results accurately.
  2. [Section 3.2 and Section 6] The AITW labels were produced by two annotators with no inter-annotator agreement reported (Section 3.2), and Section 6 concedes that synthetic speech labels are sparse and hard to have high confidence in, while noise labels carry possible ambiguity. This is load-bearing because the key evidence for Whilter's synthetic-speech detection is Table 1's F1=45.6 versus BEATs' 0.0; if the positive labels are noisy or inconsistent, that comparison does not measure true class detection. Please report inter-annotator agreement (e.g., Cohen's kappa) or a double-annotated subset, give per-split positive counts for each class, and temper the synthetic-speech claims if label reliability cannot be quantified.
  3. [Table 1 and Section 5] For foreign language and synthetic speech, the BEATs baselines (both vanilla and +TAN) are degenerate: they predict all negatives, giving FPR=0.0, FNR=100.0, and F1=0.0. Outperforming a model that never predicts the positive class is weak evidence of Whilter's advantage for speech-specific tasks. Please include non-degenerate comparisons for these classes, such as Whisper's built-in language identification for foreign language or a stronger anti-spoofing baseline than the reported AASIST variants, and discuss the collapse of the BEATs baselines explicitly in the text.
  4. [Section 4.3 and Table 1] No confidence intervals or significance tests are reported for any metric. Given the small number of synthetic-speech positives in the test set, acknowledged in Section 6, point estimates such as F1=45.6 may be highly unstable. Please add bootstrap confidence intervals or statistical tests for the principal comparisons (e.g., Whilter versus BEATs+TAN on noise and music, Whilter versus Whisper small on foreign language) so that the reader can assess whether the observed differences are reliable.
minor comments (6)
  1. [Figure 2] The y-axis label 'Occurences' is misspelled; it should be 'Occurrences'.
  2. [Section 3.2] The sentence listing the dataset sizes is missing a closing parenthesis: 'In total 18,346 samples were annotated for fine-tuning (≈ 55 hrs, 1353 for validation (≈ 4 hrs) and 1716 for testing ( ≈ 5 hrs), resulting in 21,414 samples in total ( ≈ 64 hrs).' Please rephrase for readability and correct the punctuation.
  3. [Section 4.2] The paper does not describe how the language decision is made for the Whisper foreign-language baselines; please specify whether the top-1 language token is used and what threshold, if any, is applied.
  4. [Section 4.2] The fine-tuning protocol for the AASIST baseline is not given; please include the number of epochs, learning rate, and data split used for the fine-tuned variant.
  5. [Section 2.4] The notation y ∈ R^{N×1} is slightly misleading since each y_n is a binary label; using y ∈ {0,1}^{N×1} would be clearer.
  6. [Table 1] For the 'Foreign Language' rows, Whisper (small) and Whisper (large) are not marked as FT, which is correct, but the table would benefit from a note that these baselines use Whisper's built-in language identification rather than the AITW labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model is trained on human-annotated labels, evaluated on a held-out test set, and baselines are fit on the same training split; no predicted quantity reduces by construction to a fitted input.

full rationale

This is an empirical supervised-learning paper rather than a derivation chain. Whilter is trained with binary cross-entropy on human-annotated AITW labels and on artificially mixed non-ITW labels, then evaluated on a disjoint held-out test portion of AITW. No output class is defined in terms of a fitted parameter from the same data, and no reported metric is forced by construction. The DNSMOS+LR, fine-tuned AASIST, and fine-tuned BEATs baselines are fit on the AITW fine-tuning split and evaluated on the test split, which is standard supervised evaluation rather than circular prediction. Whisper and BEATs are external pretrained models; using Whisper features is transfer learning, not a self-referential loop. The paper's self-citations to prior speech-quality and speech-enhancement work are motivational and are not load-bearing for the central claim, which rests on held-out test results. The acknowledged weaknesses -- lack of reported inter-annotator agreement, sparse and uncertain synthetic-speech labels, and ambiguity in noise labels -- are data-quality and generalization risks, not circularity. The abstract's phrasing about 'F1 scores above 85%... for three of five subtasks' is imprecise relative to Table 1, since only multispeaker and foreign-language F1 exceed 85%, but that is a reporting-accuracy concern, not a circularity concern. Overall, the evaluation is self-contained against external benchmarks and exhibits no step where a prediction is equivalent to its input by definition.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical performance claim, so the relevant free inputs are training hyperparameters and architecture choices listed above. No new physical or mathematical entities are introduced. The strongest data assumption is that the AITW annotations, produced by two annotators with no agreement metric, are accurate labels for all five classes.

free parameters (8)
  • Learning rate and schedule = 1e-5, exponential decay gamma 0.7/0.98
    Hand-chosen; no sweep documented. Applies to both training stages.
  • Training epochs = 10 simulated + 100 fine-tuning
    Hand-chosen stopping points; no early stopping.
  • Batch size and samples per epoch = batch 64, 15,000 samples/epoch
    Computational choices; not tuned.
  • Dynamic mixing ratio and SNR range = 25% of batch mixed; SNR/SSR in (-5,10) dB
    Data augmentation design choices.
  • Positive class sampling weight = #neg / #pos
    Used in weighted random sampling to address imbalance.
  • Input length = 30 s (1500 frames)
    Whisper encoder fixed input; clips padded or truncated to 30s.
  • Whisper intermediate layer weights = 12 learned scalars
    Learned weights for weighted sum of encoder layers; fitted on AITW data.
  • Prediction network architecture = 4-layer transformer, 4 heads, 256-dim; 5 attention pooling heads
    Hand-chosen capacity; no architecture search.
assumptions (5)
  • domain assumption Whisper encoder features are sufficient to discriminate the five classes
    The method relies on Whisper's frozen representations carrying the needed information (Section 2.1).
  • domain assumption AITW human annotations are accurate ground truth for all five classes
    Two annotators labeled 21k samples with no inter-annotator agreement metric (Section 3.2).
  • domain assumption Random subsets of Emilia and YODAS are representative of in-the-wild speech for TTS data filtering
    Generalization of results depends on this representativeness (Section 3.2).
  • standard math Binary cross-entropy is an appropriate loss for multi-label classification
    Standard choice; labels are not mutually exclusive (Section 2.4).
  • standard math EER, F1, precision, and recall are valid metrics for threshold-based classification
    Standard classification metrics (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification." pith.science (2026). https://pith.science/paper/BZIX3N5G

@misc{pith2026250721642,
  author       = {Pith},
  title        = {Pith review of: Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZIX3N5G}},
  note         = {Machine review of arXiv:2507.21642}
}
read the original abstract

Large-scale in-the-wild speech datasets have become more prevalent in recent years due to increased interest in models that can learn useful features from unlabelled data for tasks such as speech recognition or synthesis. These datasets often contain undesirable features, such as multiple speakers, non-target languages, and music, which may impact model learning. The Whilter model is proposed as a multitask solution to identify these undesirable samples. Whilter uses a Whisper encoder with an attention-based classifier to solve five diverse classification problems at once. In addition, an annotated dataset is published for a subset of two popular in-the-wild corpora. Whilter achieves F1 scores above 85% and equal error rates of 6.5% to 7.8% for three of five subtasks, outperforming a state-of-the-art BEATs classifier on speech-specific classes, with a notable decrease in processing time compared to a combination of single-task alternatives.

Figures

Figures reproduced from arXiv: 2507.21642 by the authors.

Figure 1
Figure 1. Diagram of the Whilter model, composed of a frozen Whisper encoder with learnable layer weights, a transformer prediction network and attention-pooling-based classification layers. model2 [13] is used for extracting rich audio features. Whisper is a multilingual ASR foundation model trained using a large￾scale weak supervision and multitask learning (MTL) [21] for speech translation, spoken language identification, … view at source ↗
Figure 2
Figure 2. Number of occurrences of each label across the entire AITW dataset as well as the total number of samples with no label. shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 43 canonical work pages

  1. [1]

    In-the-wild (ITW) refers to data that is not recorded or collected in a highly controlled environ- ment [1]

    Introduction Increasingly large corpora have been proposed in recent years from in-the-wild data sources for training deep learning speech processing models [1, 2, 3]. In-the-wild (ITW) refers to data that is not recorded or collected in a highly controlled environ- ment [1]. Rather, ITW speech datasets typically consist of audio sourced from public video...

    work page

  2. [2]

    Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification

    Whilter The Whilter model W, takes a single channel audio signal of length L, denoted x ∈ RL, and maps it to N output classes, i.e. W : 1 × L 7→ N × 1. Whilter is composed of three main components: a frozen Whisper [13] encoder network, a learnable Transformer [18] network, and a bank of atten- tion pooling layers [19] for the five output classes shown in...

    work page Pith review arXiv

  3. [3]

    For the first training stage, simple filename and label pairs are derived from non-ITW datasets

    Data Preparation A two-stage training approach using training with artificially mixed non-ITW data, followed by fine-tuning with the AITW data. For the first training stage, simple filename and label pairs are derived from non-ITW datasets. Dynamic mixing [24] , popularized for speech separation [15], is used for augment- ing the combinations of these lab...

    work page 2018

  4. [4]

    Experimental Setup 4.1. Training Configurations The model was trained using an ADAM optimizer [33] and an exponentially decaying learning rate scheduler whereby the learning rate, η, was decayed by a factor, γ, at the end of each epoch. For both training stages, 15, 000 samples are selected per epoch using a weighted random sampler. A batch size of 64 is ...

    work page

  5. [5]

    Results Results for both ST and MT approaches are shown in Tab. 1. For ST multispeaker classification, the Whilter model outper- formed the Pyannote model [10] in EER by 7.8% and F1 by 12.8%. Whilter was also faster on average than Pyannote by more than a factor of 10. Comparing Whilter to Whisper for ST foreign language classification, both have comparab...

    work page

  6. [6]

    noisiness

    Discussion This paper presents a first effort towards solving a non-trivial MT classification problem. A key finding is that speech-specific foundation models provide a clear advantage in foreign lan- guage and synthetic speech detection, while audio foundation models perform slightly better with music and noise. Future work could investigate combining sp...

    work page

  7. [7]

    It was shown that the AITW dataset can be used to train classifiers on multispeaker, foreign language, background music, noise and synthetic speech labels

    Conclusion In this work, the Whilter model was proposed along with the AITW dataset, for MT data filtering of in-the-wild speech recordings. It was shown that the AITW dataset can be used to train classifiers on multispeaker, foreign language, background music, noise and synthetic speech labels. Whilter outperformed numerous widely used open-source models...

    work page

  8. [8]

    An open-source speaker gender detection framework for moni- toring gender equality,

    D. Doukhan, J. Carrive, F. Vallet, A. Larcher, and S. Meignier, “An open-source speaker gender detection framework for moni- toring gender equality,” inICASSP 2018. IEEE, 2018

    work page 2018

Show all 46 references
  1. [9]

    Label Studio: Data labeling software,

    M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data labeling software,” 2020-2024. [Online]. Available: https://github.com/HumanSignal/label-studio

    2020

  2. [10]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” in Interspeech 2018, 2018, pp. 1086–1090

    2018

  3. [11]

    YODAS: Youtube-Oriented Dataset for Audio and Speech,

    X. Li, S. Takamichi, T. Saeki, W. Chen, S. Shiota, and S. Watan- abe, “YODAS: Youtube-Oriented Dataset for Audio and Speech,” in ASRU 2023, 2023, pp. 1–8

    2023

  4. [12]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y . Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.05361

    2024 arXiv

  5. [13]

    Speak, read and prompt: High-fidelity text-to-speech with min- imal supervision,

    E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with min- imal supervision,” Transactions of the Association for Computa- tional Linguistics, vol. 11...

    2023

  6. [14]

    synthetic speech and non-target languages in many cases)

    or source separation [15]) or discarded entirely if there is potentially corrupting information contained within them (i.e. synthetic speech and non-target languages in many cases). The motivation for using the MT approach is threefold. Firstly, due to the size of most ITW dat...

  7. [15]

    mhubert-147: A compact multilingual hubert model,

    M. Zanon Boito, V . Iyer, N. Lagos, L. Besacier, and I. Calapode- scu, “mhubert-147: A compact multilingual hubert model,” in In- terspeech 2024, 2024, pp. 3939–3943

    2024

  8. [16]

    Sentence level intelligibility eval- uation for mandarin text-to-speech systems using semantically unpredictable sentences,

    J. Li, D. Sityaev, and J. Hao, “Sentence level intelligibility eval- uation for mandarin text-to-speech systems using semantically unpredictable sentences,” in Interspeech 2007, 2007, pp. 1350– 1353

    2007

  9. [17]

    Data-filtering methods for self-training of auto- matic speech recognition systems,

    A.-L. Georgescu, C. Manolache, D. Oneat ¸ ˘a, H. Cucu, and C. Burileanu, “Data-filtering methods for self-training of auto- matic speech recognition systems,” in 2021 SLT Workshop, 2021, pp. 1–7

    2021

  10. [18]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnet...

    2017

  11. [19]

    pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,

    H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,” in Interspeech 2023, 2023

    2023

  12. [20]

    FoR: A dataset for synthetic speech detection,

    R. Reimao and V . Tzerpos, “FoR: A dataset for synthetic speech detection,” in 2019 International Conference on Speech Technol- ogy and Human-Computer Dialogue (SpeD), 2019, pp. 1–10

    2019

  13. [21]

    AASIST: Audio Anti-Spoofing Us- ing Integrated Spectro-Temporal Graph Attention Networks,

    J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “AASIST: Audio Anti-Spoofing Us- ing Integrated Spectro-Temporal Graph Attention Networks,” in ICASSP 2022, 2022, pp. 6367–6371

    2022

  14. [22]

    This as- sumption is validated later in our results, cf

    or CLAP [23], because three of the subtasks are speech- specific (multispeaker, foreign language and synthetic speech classification) and more general audio foundation models may not have a nuanced enough representation of speech. This as- sumption is validated later in our re...

  15. [23]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in ICML 2023. JMLR.org, 2023

    2023

  16. [24]

    Perceive and predict: Self-supervised speech representation based loss func- tions for speech enhancement,

    G. Close, W. Ravenscroft, T. Hain, and S. Goetze, “Perceive and predict: Self-supervised speech representation based loss func- tions for speech enhancement,” in ICASSP 2023, 2023, pp. 1–5

    2023

  17. [25]

    Transcription-free fine-tuning of speech separation models for noisy and reverberant multi-speaker automatic speech recognition,

    W. Ravenscroft, G. Close, S. Goetze, T. Hain, M. Soleymanpour, A. Chowdhury, and M. C. Fuhs, “Transcription-free fine-tuning of speech separation models for noisy and reverberant multi-speaker automatic speech recognition,” in Interspeech 2024 , 2024, pp. 4998–5002

    2024

  18. [26]

    Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate asr features and human memory models,

    R. Mogridge, G. Close, R. Sutherland, T. Hain, J. Barker, S. Goetze, and A. Ragni, “Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate asr features and human memory models,” in ICASSP 2024, 2024, pp. 306– 310

    2024

  19. [27]

    Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,

    O. Chetia Phukan, G. Kashyap, A. B. Buduru, and R. Sharma, “Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,” in NAACL 2024, Jun. 2024, pp. 2496–2506

    2024

  20. [28]

    Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,

    G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality pre- diction with crowdsourced datasets,” inInterspeech 2021. ISCA, Aug. 2021

    2021

  21. [29]

    Hallucination in perceptual metric-driven speech enhancement networks,

    G. Close, T. Hain, and S. Goetze, “Hallucination in perceptual metric-driven speech enhancement networks,” inEUSIPCO 2024, 2024, pp. 21–25

    2024

  22. [30]

    An overview of multi-task learning in deep neural networks,

    S. Ruder, “An overview of multi-task learning in deep neural networks,” 2017. [Online]. Available: https://arxiv.org/abs/1706. 05098

    2017

  23. [31]

    BEATs: Audio pre-training with acoustic to- kenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic to- kenizers,” in ICML 2023, A. Krause, E. Brunskill, K. Cho, B. En- gelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202, Jul 2023, pp. 5178–5193

    2023

  24. [32]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023, 2023, pp. 1–5

    2023

  25. [33]

    Wavesplit: End-to-end speech separation by speaker clustering,

    N. Zeghidour and D. Grangier, “Wavesplit: End-to-end speech separation by speaker clustering,” IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, p. 2840–2849, Jul. 2021

    2021

  26. [34]

    The AMI Meeting Corpus: A Pre- announcement,

    J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V . Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner, “The AMI Meeting Corpus: A Pre- announcement,” inMachine Learning for M...

    2006

  27. [35]

    M2Met: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge,

    F. Yu, S. Zhang, Y . Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu, “M2Met: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge,” in ICASSP 2022, 2022, pp. 6167–6171

    2022

  28. [36]

    MLS: A Large-Scale Multilingual Dataset for Speech Research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A Large-Scale Multilingual Dataset for Speech Research,” in Interspeech 2020, 2020, pp. 2757–2761

    2020

  29. [37]

    MUSAN: A Music, Speech, and Noise Corpus,

    D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” 2015, arXiv:1510.08484v1

    2015 arXiv

  30. [38]

    An open dataset of synthetic speech,

    A. Yaroshchuk, C. Papastergiopoulos, L. Cuccovillo, P. Aichroth, K. V otis, and D. Tzovaras, “An open dataset of synthetic speech,” in IEEE WIFS 2023, 2023, pp. 1–6

    2023

  31. [39]

    OpenMIC-2018: An Open Dataset for Multiple Instrument Recognition,

    E. J. Humphrey, S. Durand, and B. McFee, “OpenMIC-2018: An Open Dataset for Multiple Instrument Recognition,” inISMIR 2018, 2018

    2018

  32. [40]

    Learning sound event classifiers from web audio with noisy labels,

    E. Fonseca, M. Plakal, D. P. W. Ellis, F. Font, X. Favory, and X. Serra, “Learning sound event classifiers from web audio with noisy labels,” in ICASSP 2019, 2019, pp. 21–25

    2019

  33. [41]

    The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings,

    J. Thiemann, N. Ito, and E. Vincent, “The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings,” in21st Interna- tional Congress on Acoustics, Jun. 2013

    2013

  34. [42]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” in ICLR 2015, May 2015

    2015

  35. [43]

    Data augmen- tation for speech separation,

    A. Alex, L. Wang, P. Gastaldo, and A. Cavallaro, “Data augmen- tation for speech separation,”Speech Communication, vol. 152, p. 102949, 2023

    2023

  36. [44]

    Dnsmos: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. A. Reddy, V . Gopal, and R. Cutler, “Dnsmos: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2021, 2021, pp. 6493–6497

    2021

  37. [45]

    Convolutional recurrent neural networks for poly- phonic sound event detection,

    E. Cakir, G. Parascandolo, T. Heittola, H. Huttunen, T. Virta- nen, E. Cakir, G. Parascandolo, T. Heittola, H. Huttunen, and T. Virtanen, “Convolutional recurrent neural networks for poly- phonic sound event detection,” IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 25, n...

    2017

  38. [46]

    Mlaad: The multi- language audio anti-spoofing dataset,

    N. M. M ¨uller, P. Kawa, W. H. Choong, E. Casanova, E. G ¨olge, T. M¨uller, P. Syga, P. Sperl, and K. B¨ottinger, “Mlaad: The multi- language audio anti-spoofing dataset,” in IJCNN 2024, 2024, pp. 1–7

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.