Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OpenBEATs, an open-source reimplementation and scale-up of the BEATs audio encoder, achieves state-of-the-art results across bioacoustics, environmental sound, and audio reasoning benchmarks while using about a quarter of the parameters…

desk verdict OpenBEATs is a genuinely valuable open-source release—the first reproducible BEATs pretraining pipeline plus a broad evaluation suite—but the central claim that multi-domain pretraining drives the gains is unsupported because data hours, batch size, and model scale all change at once. read the letter →

arxiv 2507.14129 v2 pith:USTVYRFE submitted 2025-07-18 cs.SD eess.AS

Shikhar Bharadwaj , Samuele Cornell , Kwanghee Choi , Satoru Fukayama , Hye-jin Shim , Soham Deshmukh , Shinji Watanabe This is my paper · ORCID
classification cs.SDeess.AS
keywords OpenBEATsaudioencodermaskedtokenpredictionself-supervisedlearningmulti-domainpre-trainingbioacousticsreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make masked token prediction, the pre-training objective behind BEATs, openly reproducible and to show that it works as a general-purpose audio encoder when scaled across domains. It reimplements the BEATs pipeline from scratch, expands pre-training from 5.8k hours of AudioSet to 20k hours spanning music, environmental sound, and bioacoustics, and scales the model to 300M parameters. On a 25-dataset, six-task evaluation the resulting OpenBEATs models reach state-of-the-art results on six bioacoustics datasets, two environmental sound datasets, and five reasoning datasets, matching or beating models with over a billion parameters at one-fourth their size. The broader claim a sympathetic reader should take away is that multi-domain masked token prediction produces audio representations that transfer across traditional classification, detection, and open-ended semantic reasoning tasks.

What carries the argument

The carrying mechanism is BEATs' two-stage iterative pre-training: a transformer encoder predicts discrete tokens for masked 16x16 mel-spectrogram patches (masked language modeling over audio), while a teacher-student tokenizer with a learned codebook quantizes audio into token indices via knowledge distillation, and the two stages are refined alternately through several iterations. OpenBEATs keeps this mechanism but changes the input diet and scale: it mixes music (FMA), environmental sound (AudioSet, FreeSound, BBC Sound Effects), and bioacoustics (iNaturalist) into 20k hours, trains with variable-length input sequences, and scales the encoder from 90M to 300M parameters using a base/large design. The multi-domain mixture is what the paper credits for cross-domain transfer, and the open implementation is what makes the mechanism independently reproducible.

What would settle it

Train OpenBEATs-Large twice with identical 20k hours, 300M parameters, batch size, and update count, once on the full multi-domain mix and once on AudioSet alone, then evaluate both on the same six task types; if the AudioSet-only model matches or beats the multi-domain model, the paper's core attribution to multi-domain pre-training is falsified.

Watch

Extended reading notes

Core claim

The paper claims that OpenBEATs, an open-source reimplementation of BEATs trained on a multi-domain mix of 20k hours and scaled to 300M parameters, learns general-purpose audio representations. It reports state-of-the-art accuracy on six of ten BEANS bioacoustics tasks, top scores on DESED and UrbanSound8K among linear-probe sound tasks, and best results on audio question answering and audio entailment, while matching Dasheng-1.2B on several benchmarks despite using a quarter of the parameters and far less pre-training data. The paper argues that these results show token-prediction masked modeling, when combined with multi-domain data, transfers across domains and even outperforms audio-text contrastive baselines that use paired text supervision. It also releases the complete pre-training and evaluation code, checkpoints, and logs to make the recipe auditable and reusable.

Load-bearing premise

The paper attributes the accuracy gains to multi-domain pre-training, but it never trains a matched model on a single domain with the same total hours, parameter count, and number of updates, so the gains might come from more data or a bigger model rather than from domain diversity.

Editorial extensions

If this is right

  • A 300M-parameter masked-token audio encoder can match or outperform billion-parameter models, suggesting this objective scales efficiently in both parameters and data.
  • Multi-domain pre-training on music, environmental sound, and bioacoustics transfers to semantic reasoning tasks such as audio question answering and entailment, not just classification.
  • Bioacoustics no longer requires in-domain or text-supervised encoders; a general-purpose SSL encoder trained on a broad mix achieves top results on most BEANS datasets.
  • Fully open pre-training and evaluation code, checkpoints, and logs allows any group to reproduce the exact training recipe and to audit the reported numbers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Mixing ratios among music, environmental, and bioacoustic data are likely a tunable ingredient, so ablating proportions of each domain would map which data sources drive each downstream gain.
  • Because the paper excludes speech on the grounds that a 175 ms patch is too coarse for phoneme timing, the same recipe could plausibly cover speech with finer patch resolution or a separate tokenizer.
  • OpenBEATs' reasoning-task gains suggest audio encoders for audio-language models may benefit more from broad semantic pre-training than from scale alone, a hypothesis the released pipeline makes directly testable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes OpenBEATs, an open-source reimplementation and extension of the BEATs masked-token-prediction audio encoder. The authors scale BEATs pre-training from AudioSet-only 5.8k hours to 20k hours by adding music, environmental-sound, and bioacoustics corpora, and scale the model from 90M to 300M parameters. They release pre-training and evaluation code, checkpoints, data preprocessing scripts, and training logs. The encoder is evaluated with linear probing on X-ARES sound tasks, full fine-tuning on environmental sound and BEANS bioacoustics tasks, audio reasoning tasks (entailment, question answering, captioning), and music classification. The paper reports state-of-the-art or competitive results against Dasheng-1.2B, BEATs, Audio-MAE, and audio-text contrastive models, and interprets these results as evidence that multi-domain masked token prediction yields general-purpose audio representations.

Significance. If taken at face value, the paper is a useful empirical contribution: it demonstrates that a fully open-source BEATs recipe can be trained on 20k hours of heterogeneous audio and reach or beat much larger models on several benchmarks. The release of the training pipeline, checkpoints, logs, and evaluation suite is a genuine service to the community and addresses a longstanding reproducibility gap around BEATs. The multi-domain interpretation, however, is not currently cleanly supported: the experimental design changes data volume, batch size, model scale, and learning rate together with domain diversity, and the paper itself documents saturation and scale effects. The headline attribution to domain diversity therefore needs an ablated control or substantially softened claims.

major comments (4)
  1. [Section 3, Table 1, Section 5] No matched AudioSet-only control isolates the multi-domain effect. Table 1 varies data hours (5.8k to 20k), batch size (4.6k to 10.7k seconds), and, for the Large model, parameter count and learning rate simultaneously with the domain expansion. Section 5 explicitly reports that the 90M model saturates and that scaling to 300M pays off, so data volume and model scale are demonstrably active variables. The abstract and conclusion attribute the gains to 'multi-domain datasets' and 'multi-domain pre-training,' but every headline result could plausibly be explained by data volume, batch size, or model capacity. Please train an AudioSet-only OpenBEATs with matched hours, batch size, and parameter count (or otherwise vary the domain composition while holding these factors fixed), or reframe the contribution as scaled open-source BEATs without the multi-domain attribution.
  2. [Table 6, Section 5] The claim that OpenBEATs 'outperforms the text-supervised audio-text contrastive baselines like LAION-CLAP and BioLingual' is not supported by Table 6. On the displayed BEANS tasks, BioLingual remains higher than OpenBEATs Large on Watkins (89.4 vs 88.2), CBI (74.4 vs 69.4), Dogs (97.1 vs 95.7), Rfcx (47.5 vs 46.2), Gibbons (17.8 vs 14.9), and Enabirds (67.7 vs 66.0), and the column averages are essentially equal. The sentence should be revised to say that OpenBEATs is competitive with, or better than, these baselines on selected tasks, or the comparison should be restricted to SSL-only models.
  3. [Table 5, Section 5] The caption 'OpenBEATs achieves best performance on environmental sound detection tasks' is not supported by the table as printed. On AS-2M, OpenBEATs Large's 42.1 mAP is below Audio-MAE's 47.3/47.4 and below the BEATs-reported 48.0; against BEATs' own copy of AudioSet, the margin is only 0.5 mAP (42.1 vs 41.6). The paper attributes most of the AudioSet gap to the different YouTube copy, but to support a SOTA claim all baselines should be evaluated and reported on the same AudioSet copy, or the claim should be qualified as 'best among models evaluated on our copy.'
  4. [Tables 4-7] Several close SOTA margins are reported without error bars, repeated runs, or significance tests (e.g., Table 4 DSD 0.57 vs 0.56; Table 5 AS-2M 42.1 vs 41.6; Table 6 Rfcx 46.2 vs 46.4). Given the breadth of the SOTA claims, the close comparisons need either variance information or softer 'competitive' phrasing.
minor comments (3)
  1. [Abstract, Table 3] The abstract states 'twenty five datasets' and 'five reasoning datasets,' but Table 3 lists three reasoning datasets and the named entries sum to 24 when the ten BEANS tasks are counted individually; please reconcile these counts.
  2. [Section 3, Figure 1] There is a typo in Section 3 ('real-wold' should be 'real-world'), and the Figure 1 caption contains garbled symbols such as '♂¶usic' and ' /volume-down', likely an encoding issue.
  3. [Table 5] The table note says '†' denotes values reported by BEATs, but Audio-MAE values also come from the BEATs paper; clarify the provenance for each baseline so readers know which numbers were independently reproduced.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central results are external benchmark measurements; the multi-domain attribution is a confound, not a circular derivation.

full rationale

The paper's derivation chain is a standard empirical pipeline: pretrain on a multi-domain corpus, then fine-tune or linearly probe on held-out benchmarks. No fitted parameter is renamed as a prediction, no equation reduces to its own input, and no load-bearing claim rests on a self-citation chain. The central comparisons (BEANS, ESC-50, reasoning tasks, music tasks) use external datasets not in the pretraining mix, and the gains over BEATs and Dasheng are measured rather than constructed. The paper does contain minor self-referential elements: AudioSet appears in both pretraining and evaluation, and the AudioSet evaluation copy is self-built with a reported 5% difference from the original. These are comparability or contamination issues, not circularity under the defined patterns. The missing AudioSet-only control with matched hours and parameters is a confound between domain diversity and data/scale, but that is an experimental-design limitation, not a circular derivation. Therefore, no circular step is identified, and the score reflects only the minor self-referential evaluation overlap.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The main load-bearing choices are inherited architecture settings, hand-set hyperparameters, and assumptions about data representativeness and protocol comparability.

free parameters (4)
  • OpenBEATs-Large architecture scale = 300M params, hidden 1024, FFN 4096, 24 layers, 16 heads
    Hand-chosen following the HuBERT base/large pattern; downstream performance depends on it and no sensitivity study is reported.
  • Pre-training schedule = 400k updates, 40k warmup
    Adopted from BEATs; no evidence it is optimal for the 20k-hour mixed corpus.
  • Learning rate = Base 5e-4, Large 1e-4
    Hand-selected settings; no learning-rate exploration is reported.
  • Acoustic tokenizer codebook size = 1024
    Inherited from BEATs and not varied in this work.
assumptions (4)
  • domain assumption Masked token prediction with a learned acoustic tokenizer is a valid general-purpose pretraining objective across environmental sound, bioacoustics, and music.
    This is the core methodological premise, inherited from BEATs and tested only empirically here.
  • domain assumption The BEATs patch layout (16x16 patches over 128 mel bins at 16 kHz) is adequate for music and bioacoustics.
    Section 3 excludes speech for temporal-resolution reasons but does not adapt patch size for the new domains.
  • domain assumption The combined pretraining corpora are representative of the evaluation domains and are free of problematic overlap.
    Data comes from FMA, AudioSet, FreeSound, BBC, iNaturalist, and others, but no domain-balance or contamination analysis is given.
  • domain assumption Downstream evaluation protocols from BEANS, X-ARES, and prior recipes are implemented faithfully and are comparable across baselines.
    Many baseline numbers are re-implemented or taken from prior work; there is no statistical validation of protocol parity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder." pith.science (2026). https://pith.science/paper/USTVYRFE

@misc{pith2026250714129,
  author       = {Pith},
  title        = {Pith review of: OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USTVYRFE}},
  note         = {Machine review of arXiv:2507.14129}
}
read the original abstract

Masked token prediction has emerged as a powerful pre-training objective across language, vision, and speech, offering the potential to unify these diverse modalities through a single pre-training task. However, its application for general audio understanding remains underexplored, with BEATs being the only notable example. BEATs has seen limited modifications due to the absence of open-source pre-training code. Furthermore, BEATs was trained only on AudioSet, restricting its broader downstream applicability. To address these gaps, we present OpenBEATs, an open-source framework that extends BEATs via multi-domain audio pre-training. We conduct comprehensive evaluations across six types of tasks, twenty five datasets, and three audio domains, including audio reasoning tasks such as audio question answering, entailment, and captioning. OpenBEATs achieves state-of-the-art performance on six bioacoustics datasets, two environmental sound datasets and five reasoning datasets, performing better than models exceeding a billion parameters at one-fourth their parameter size. These results demonstrate the effectiveness of multi-domain datasets and masked token prediction task to learn general-purpose audio representations. To promote further research and reproducibility, we release all pre-training and evaluation code, pretrained and fine-tuned checkpoints, and training logs at https://github.com/Shikhar-S/OpenBEATs

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    eess.AS 2025-11 conditional novelty 6.0 of 10

    A 10.7M-pair audio-caption corpus and systematic comparison show contrastive pretraining is more data-efficient while captioning scales better, and supervised initialization yields diminishing returns.

Reference graph

Works this paper leans on

54 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    BEATs: audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: audio pre-training with acoustic tokenizers,” in ICML, 2023

    work page 2023

  2. [2]

    SSAST: Self-supervised audio spectrogram transformer,

    Y . Gong, C.-I. Lai, Y .-A. Chung, and J. Glass, “SSAST: Self-supervised audio spectrogram transformer,” in AAAI, 2022

    work page 2022

  3. [3]

    MAE-AST: Masked autoencoding audio spectrogram transformer,

    A. Baade, P. Peng, and D. Harwath, “MAE-AST: Masked autoencoding audio spectrogram transformer,” Interspeech, 2022

    work page 2022

  4. [4]

    Masked autoencoders that listen,

    P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” NeurIPS, 2022

    work page 2022

  5. [5]

    DCASE 2024 task 4: Sound event detection with heterogeneous data and missing labels,

    S. Cornell, J. Ebbers, C. Douwes, I. Mart´ın-Morat´o, M. Harju, A. Mesaros, and R. Serizel, “DCASE 2024 task 4: Sound event detection with heterogeneous data and missing labels,” DCASE Workshop, 2024

    work page 2024

  6. [6]

    BEATs-based audio captioning model with instructor embedding supervision and chatgpt mix-up,

    S.-L. Wu, X. Chang, G. Wichern, J.-w. Jung, F. Germain, J. L. Roux, and S. Watanabe, “BEATs-based audio captioning model with instructor embedding supervision and chatgpt mix-up,” DCASE2023 Challenge, Tech. Rep., 2023

    work page 2023

  7. [7]

    Few-shot bioacoustic event detection using beats,

    F. Gelderblom, B. Cretois, P. Johnsen, F. Remonato, and T. A. Reinen, “Few-shot bioacoustic event detection using beats,” DCASE2023 Chal- lenge, Tech. Rep., 2023

    work page 2023

  8. [8]

    Open implemen- tation and study of best-rq for speech processing,

    R. Whetten, T. Parcollet, M. Dinarelli, and Y . Est `eve, “Open implemen- tation and study of best-rq for speech processing,” ICASSP workshop on Self-supervision in Audio, Speech and Beyond , 2024

    work page 2024

Show all 54 references
  1. [9]

    Reproducing whisper-style training using an open-source toolkit and publicly available data,

    Peng et al, “Reproducing whisper-style training using an open-source toolkit and publicly available data,” in ASRU, 2023

    2023

  2. [10]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019

    1904 arXiv

  3. [11]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  4. [12]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduch- intala, and T. Ochiai, “ESPnet: End-to-end speech processing toolkit,” in Interspeech, 2018

    2018

  5. [13]

    VERSA: A versatile evaluation toolkit for speech, audio, and music,

    Shi et al, “VERSA: A versatile evaluation toolkit for speech, audio, and music,” arXiv preprint arXiv:2412.17667 , 2024

    2024 arXiv

  6. [14]

    Scaling bioacoustic signal pre-training with million samples via mask-modeling,

    X. Deng, T. Wan, K. Xu, T. Gao, P. Qiao, D. Feng, and Y . Dou, “Scaling bioacoustic signal pre-training with million samples via mask-modeling,” in ICASSP, 2025

    2025

  7. [15]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . e. a. Li, “Mert: Acoustic music understanding model with large-scale self-supervised training,” in ICLR, 2024

    2024

  8. [16]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. e. a. Chen, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  9. [17]

    Google USM: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhanget al., “Google USM: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037 , 2023

    2023 arXiv

  10. [18]

    Towards robust speech representation learning for thousands of languages,

    W. Chen, W. Zhang, Y . Peng, X. Li, J. Tian, J. Shi, X. Chang, S. Maiti, K. Livescu, and S. Watanabe, “Towards robust speech representation learning for thousands of languages,” in EMNLP, 2024

    2024

  11. [19]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” NeurIPS, 2023

    2023

  12. [20]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, M. Zejun, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” in ICLR, 2023

    2023

  13. [21]

    GAMA: A large audio- language model with advanced audio understanding and complex reasoning abilities,

    S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “GAMA: A large audio- language model with advanced audio understanding and complex reasoning abilities,” EMNLP, 2024

    2024

  14. [22]

    Mellow: a small audio language model for reasoning,

    S. Deshmukh, S. Dixit, R. Singh, and B. Raj, “Mellow: a small audio language model for reasoning,” arXiv preprint arXiv:2503.08540 , 2025

    2025 arXiv

  15. [23]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in ICASSP, 2017

    2017

  16. [24]

    ESC: Dataset for environmental sound classification,

    K. J. Piczak, “ESC: Dataset for environmental sound classification,” in Proceedings of ACM international conference on Multimedia , 2015

    2015

  17. [25]

    Scaling up masked audio encoder learning for general audio classification,

    H. Dinkel, Z. Yan, Y . Wang, J. Zhang, Y . Wang, and B. Wang, “Scaling up masked audio encoder learning for general audio classification,” Interspeech, 2024

    2024

  18. [26]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    D. Alexey, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2020

    2020

  19. [27]

    Self-supervised learning with random-projection quantizer for speech recognition,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in ICML, 2022

    2022

  20. [28]

    Neural discrete representation learning,

    A. van den Oord, O. Vinyals, and k. kavukcuoglu, “Neural discrete representation learning,” in Advances in Neural Information Processing Systems, vol. 30, 2017

    2017

  21. [29]

    FMA: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in International Society for Music Information Retrieval Conference, 2017

    2017

  22. [30]

    The INaturalist species classification and detection dataset,

    V . H. et al., “The INaturalist species classification and detection dataset,” in CVPR, 2018

    2018

  23. [31]

    Transferable models for bioacoustics with human language supervision,

    D. Robinson, A. Robinson, and L. Akrapongpisak, “Transferable models for bioacoustics with human language supervision,” in ICASSP, 2024

    2024

  24. [32]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,

    Mei et al, “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    2024

  25. [33]

    CochlScene: Acquisition of acoustic scene data using crowdsourcing,

    I.-Y . Jeong and J. Park, “CochlScene: Acquisition of acoustic scene data using crowdsourcing,” in Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) . IEEE, 2022

    2022

  26. [34]

    The epic-kitchens dataset: Collection, challenges and baselines,

    D. Damen, H. Doughty et al. , “The epic-kitchens dataset: Collection, challenges and baselines,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 11, pp. 4125–4141, 2020

    2020

  27. [35]

    Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,

    N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” in DCASE Workshop, 2019

    2019

  28. [36]

    A dataset and taxonomy for urban sound research,

    J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in Proceedings of ACM international conference on Multimedia, 2014

    2014

  29. [37]

    FSD50K: An open dataset of human-labeled sound events,

    E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, pp. 829–852, 2022

    2022

  30. [38]

    General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline,

    E. Fonseca, M. Plakal, F. Font, D. P. Ellis, X. Favory, J. Pons, and X. Serra, “General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline,” in DCASE Workshop, 2023

    2023

  31. [39]

    Clotho: An audio captioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” in ICASSP, 2020

    2020

  32. [40]

    BEANS: The benchmark of animal sounds,

    M. Hagiwara, B. Hoffman, J.-Y . Liu, M. Cusimano, F. Effenberger, and K. Zacarian, “BEANS: The benchmark of animal sounds,” in ICASSP, 2023

    2023

  33. [41]

    Audio entailment: Assessing deductive reasoning for audio understanding,

    S. Deshmukh, S. Han, H. Bukhari, B. Elizalde, H. Gamper, R. Singh, and B. Raj, “Audio entailment: Assessing deductive reasoning for audio understanding,” in AAAI, 2025

    2025

  34. [42]

    Clotho-aqa: A crowdsourced dataset for audio question answering,

    S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho-aqa: A crowdsourced dataset for audio question answering,” in EUSIPCO, 2022

    2022

  35. [43]

    CIDEr: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “CIDEr: Consensus- based image description evaluation,” in CVPR, 2015, pp. 4566–4575

    2015

  36. [44]

    Music genre clas- sification using transfer learning on log-based mel spectrogram,

    J. Mehta, D. Gandhi, G. Thakur, and P. Kanani, “Music genre clas- sification using transfer learning on log-based mel spectrogram,” in 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), 2021

    2021

  37. [45]

    The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,

    B. L. Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,” arXiv preprint arXiv:1306.1461 , 2013

    2013 arXiv

  38. [46]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” in ICML, 2017

    2017

  39. [47]

    The ICME 2025 audio encoder capability challenge,

    J. Zhang, H. Dinkel, Q. Song, H. Wang, Y . Niu, S. Cheng, X. Xin, K. Li, W. Wang, Y . Wang et al. , “The ICME 2025 audio encoder capability challenge,” arXiv preprint arXiv:2501.15302 , 2025

    2025 arXiv

  40. [48]

    Data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in ICML, 2022

    2022

  41. [49]

    Robust speech recognition via large- scale weak supervision,

    A. Radford, J. W. Kim, and et al, “Robust speech recognition via large- scale weak supervision,” in ICML, 2023

    2023

  42. [50]

    EAT: self-supervised pre-training with efficient audio transformer,

    W. Chen, Y . Liang, Z. Ma, Z. Zheng, and X. Chen, “EAT: self-supervised pre-training with efficient audio transformer,” in Proceedings of the International Joint Conference on Artificial Intelligence , 2024

    2024

  43. [51]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP, 2023

    2023

  44. [52]

    A VES: Animal vocalization encoder based on self- supervision,

    M. Hagiwara, “A VES: Animal vocalization encoder based on self- supervision,” in ICASSP, 2023

    2023

  45. [53]

    Towards learning a universal non- semantic representation of speech,

    J. Shor, A. Jansen, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, and D. Emanuel, “Towards learning a universal non- semantic representation of speech,” in Proc. Interspeech, no. 2020, 2020

    2020

  46. [54]

    Automatic audio captioning with encoder fusion, multi-layer aggregation, and large language model enriched summarization,

    J.-w. Jung, D. Zhang, H. C.-H. Yang, S.-L. Wu, D. M. Chan, Z. Kong, D. Ruifan, Z. Yaqian, V . Rafael, and S. Watanabe, “Automatic audio captioning with encoder fusion, multi-layer aggregation, and large language model enriched summarization,” DCASE2024 Challenge, Tech. Rep., 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.