Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Audio-language pretraining, trained on a 10.7M-caption corpus, produces competitive transferable audio representations across speech, music, and environmental sound.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:03 UTC pith:EPFNWQJ4

load-bearing objection Useful empirical study with a real dataset contribution; the scaling conclusions need the subset sampling details before they can be fully trusted. the 4 major comments →

arxiv 2511.16757 v2 pith:EPFNWQJ4 submitted 2025-11-20 eess.AS cs.AI

Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

Wei-Cheng Tseng , Xuanru Zhou , Mingyue Huo , Yiwen Shao , Hao Zhang , Dong Yu This is my paper
classification eess.AS cs.AI
keywords audio-language pretrainingcontrastive learningcaptioning objectivegeneral-purpose audio representationCaptionStew datasetdata scalingaudio-text corporatransfer learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that audio-language pretraining—aligning audio with text captions—can build encoders that transfer across speech, music, and environmental sound, a role currently dominated by self-supervised and supervised models. It introduces CaptionStew, a 10.7M-pair corpus aggregating eight existing audio-text datasets, and uses it to compare two pretraining objectives head-to-head: contrastive alignment and generative captioning. The central results are a trade-off: contrastive learning is more data-efficient at small scale, while captioning scales better and matches or exceeds contrastive on language-involved tasks at 10M pairs. A third finding challenges common practice: initializing from an AudioSet-trained classifier helps at small scale but its benefit fades as caption data grows. A sympathetic reader would take away that objective choice should depend on data budget and target task, and that diverse caption data alone can carry a general-purpose audio encoder.

Core claim

The core discovery is that audio-language representations are competitive with—and in some tasks superior to—specialized supervised and self-supervised encoders, once the caption corpus is large and diverse enough. On speaker identification, music tagging, and audio-text retrieval, the best contrastively trained model matches or beats the AudioSet-initialized AudioSet classifier and domain-specific SSL models. The paper also documents a systematic divergence between objectives: contrastive learning yields linearly separable clip-level embeddings with high data efficiency, whereas captioning yields frame-level representations that need learned pooling and show stronger scaling on language-inv

What carries the argument

The load-bearing mechanism is the pairing of a two-tower architecture with two alternative alignment losses on a single aggregated corpus. CaptionStew (CS10M) merges eight open-source audio-text datasets spanning environmental events, music, and expressive speech, consolidating multiple captions per audio file and filtering out benchmark-overlapping samples. The contrastive objective is symmetric InfoNCE that pulls paired audio-text embeddings together; the captioning objective is a mixed autoregressive/parallel token-prediction loss (adapted from CapPa) that forces the audio encoder to supply all information needed to generate the caption. The audio encoder is a Zipformer-M backbone, chosen

Load-bearing premise

The scaling curves assume the 400K/1M/4M subsets are representative random strata of the full 10M CaptionStew corpus, but the paper never describes how those subsets were sampled.

What would settle it

Check the subset composition: if, say, the 1M subset is dominated by AudioSetCaps captions while the 10M set includes a large share of JamendoMaxCaps and ParaSpeechCaps, rerun the scaling curves on randomized stratified subsets. If the contrastive-to-captioning ordering on language tasks flips or becomes non-monotonic across draws, the scalability claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Audio-language pretraining is a viable route to general-purpose audio encoders, not just retrieval models.
  • Practitioners should choose the objective by budget: contrastive for limited data, captioning when scaling toward language-centric audio understanding.
  • Supervised AudioSet initialization can be skipped or short-circuited at large data scales without losing performance.
  • Caption diversity, not just volume, determines which downstream attributes can be learned; tasks requiring rare attributes (emotion, instrument identity) scale more slowly.
  • The 10.7M CaptionStew corpus and released recipes/models provide a foundation for larger-scale ALP experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If captioning's scaling trend holds beyond 10M pairs, generative alignment could become the default for audio encoders intended to feed LLM-based audio assistants.
  • The subset-sampling ambiguity in Section 4.5 is testable: random stratified subsets should be drawn to confirm that contrastive-vs-captioning ordering reflects scale, not dataset composition.
  • The low Distinct-n of music and speech captions suggests a bottleneck: simply aggregating existing corpora will not yield the lexical diversity needed for fine-grained acoustic attributes; new caption-generation pipelines targeting timbre, prosody, and harmony may produce larger gains than adding more of the same captions.
  • The same two-objective comparison could be run on larger backbones (e.g., transformer encoders beyond Zipformer) to see whether the data-efficiency/scalability trade-off is architecture-dependent.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper revisits audio-language pretraining (ALP) for general-purpose audio representation learning. The authors construct CaptionStew, a 10.7M audio-text pair corpus aggregated from eight existing datasets spanning environmental sound, speech, and music. They pretrain Zipformer-M audio encoders with two objectives—contrastive (Eq. 1) and captioning (Eqs. 2–3)—either from scratch or from an AudioSet-initialized checkpoint, and evaluate frozen representations on linear probing, audio-language alignment, captioning, retrieval, and open-form QA benchmarks. The main claims are that ALP yields competitive, transferable representations; that contrastive learning is more data-efficient while captioning scales better; and that supervised AudioSet initialization provides diminishing returns at larger data scales. They release data preparation recipes, training scripts, and pretrained models.

Significance. If the claims hold, this is a useful empirical contribution: it provides a large public audio-text corpus, a broad evaluation across speech/music/environmental tasks, and a systematic comparison of two ALP objectives that the field has not previously had. The paper is honest about its limitations in Appendix A.1, which strengthens credibility. However, the central quantitative conclusions rest on scaling experiments whose sampling protocol is not described, and the main evaluation omits direct comparisons to recent general-purpose ALP models (e.g., CLAP, M2D-CLAP, M2D2). These gaps must be addressed before the headline trade-off claims can be accepted.

major comments (4)
  1. [Sec. 4.5, Fig. 2] The data-scaling experiments use CaptionStew subsets at 400K, 1M, 4M, and 10M, but the paper never states how these subsets were sampled from the full corpus. The subsets could have been formed by source, by caption style, by simple concatenation order, or by random stratified sampling, and the resulting scaling curves would differ. The claim that 'contrastive is data-efficient while captioning scales better' is not identifiable without this information. Please specify the sampling algorithm, report the per-source composition of each subset, and (ideally) repeat the scaling curves with multiple strata or seeds.
  2. [Sec. 4.3, Table 3] The central claim that ALP produces 'competitive, transferable representations' is evaluated against SSL models (BEATs, wav2vec 2.0, MERT) and a supervised Zipformer-AEC baseline, but not against existing ALP encoders such as CLAP, M2D-CLAP, or M2D2, despite these being the most direct competitors. The SOTA column is quoted from other papers and involves different training/evaluation conditions. Please add direct linear-probing, retrieval, and QA results for at least one or two representative ALP encoders under the same frozen-representation protocol; otherwise the 'competitive' claim is not established relative to the current ALP literature.
  3. [Table 3 and Fig. 2] No error bars, standard deviations, or number of seeds are reported for any pretraining or downstream result. The scaling claims—particularly 'captioning scales better' and 'initialization benefits diminish at scale'—are quantitative trends that could fall within run-to-run variance, especially given that many task differences in Table 3 are small (e.g., 0.406 vs. 0.410 mAP). Please report mean and variance over at least three seeds, or explicitly justify why single-run results are sufficient. This is essential for the scaling-slope comparison.
  4. [Sec. 4.5, Fig. 2; Sec. 4.4] The conclusion that 'captioning exhibits better scalability' is based in part on downstream captioning and QA tasks that are close to the captioning pretraining objective, so the comparison is confounded by task-objective alignment. On linear-probing tasks, the paper itself states that captioning would need hundreds of millions of pairs to catch up. Please report the scaling comparison separately for objective-neutral tasks versus language-involved tasks, and temper the abstract's unqualified 'captioning exhibits better scalability' claim accordingly.
minor comments (4)
  1. [Global] There are several typos: 'preperation' (Introduction), 'disctrators' (Sec. 4.3), 'or language-involved tasks' (Sec. 4.5), and 'our results suggests' (Introduction). The notation 'CaptionStew (denoted by CS10M)' is used inconsistently; choose one short name.
  2. [Sec. 3 / Appendix A.2] The paper says overlapping samples are identified and consolidated, and that samples overlapping downstream benchmarks are filtered, but no counts or intersection statistics are given. Please report how many audio files were deduplicated and how many were removed for leakage.
  3. [Table 3] The footnote symbols (†, ‡, ††, ‡‡) are dense and hard to follow; some rows are missing values with no explicit 'not reported'. Consider splitting Table 3 into separate tables for linear probing and language/QA results to improve readability.
  4. [Appendix A.1] The authors candidly list limitations (no extensive quality control, limited technical novelty, modest scale). These are appropriate and should be kept. The dataset quality point in particular should be mentioned in the main text where CaptionStew is introduced, not only in the appendix.

Circularity Check

0 steps flagged

No circularity found; the paper is an empirical benchmark study whose comparisons are self-contained.

full rationale

This is an empirical study rather than a derivation, and I found no step where a claimed result is equivalent to its inputs by construction. The contrastive and captioning objectives are standard, independently defined losses (Eqs. 1–3), and the evaluation uses frozen encoders probed on held-out standard benchmarks after explicit overlap filtering. The headline claims—that ALP yields transferable representations, that contrastive learning is more data-efficient while captioning scales better, and that AudioSet initialization gives diminishing returns—are inductive generalizations from measured curves, not tautologies or renamed fits. The paper even acknowledges limited technical novelty and the empirical nature of its contributions in Appendix A.1. The underspecified construction of the 400K/1M/4M CaptionStew subsets in Sec. 4.5 is a legitimate threat to the identifiability of the scaling comparison, but it concerns sampling and experimental design rather than circular reasoning: the reported numbers are not defined in terms of the conclusion. Self-citations are minor (e.g., a Dynamic-Superb benchmark reference by an overlapping author) and are not load-bearing. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims do not rest on a mathematical derivation; they rest on data quality, evaluation protocol validity, subset representativeness, and single-backbone generality. These are domain assumptions rather than standard mathematical axioms.

free parameters (3)
  • Learnable contrastive temperature τ (Eq. 1) = not reported (learned)
    Standard in InfoNCE; scales logits and affects alignment strength. No benchmark-specific tuning is reported, but the learned value is not disclosed.
  • Mixing ratio of autoregressive vs parallel decoding in captioning objective = not specified
    A random fraction of each minibatch uses parallel decoding; the fraction is a hand-set hyperparameter that could influence captioning scaling results.
  • Training step budgets (600k scratch / 200k init) = 600k/200k steps
    Fixed budgets chosen by hand; objective comparisons at 10M pairs may reflect convergence differences rather than objective scalability alone.
axioms (5)
  • domain assumption Aggregated captions, unverified by humans, provide sufficient semantic supervision for general-purpose audio representation.
    Appendix A.1 explicitly states no extensive quality control is performed; the viability conclusions depend on caption fidelity and diversity.
  • domain assumption Frozen-encoder linear probing, LiT alignment, and adaptor QA are faithful measures of general-purpose representation quality.
    Section 4.2: all conclusions about representation quality are drawn through these three protocols; different protocols could change objective rankings.
  • domain assumption The 400K/1M/4M CaptionStew subsets are representative samples of the full corpus.
    Section 4.5: the controlled scaling experiments depend on subset representativeness, but the sampling procedure is not described.
  • domain assumption Zipformer-M is a sufficient general-purpose backbone for comparing audio-language objectives.
    Appendix A.3 provides preliminary evidence on a few tasks, but all objective comparisons use a single backbone; rankings may not generalize to other encoders.
  • domain assumption Removing samples overlapping downstream benchmarks prevents data leakage.
    Section 3 and A.2 list filtered benchmarks, but the overlap-detection method is not detailed; leakage would inflate transfer scores.

pith-pipeline@v1.3.0-alltime-deepseek · 20243 in / 10947 out tokens · 96482 ms · 2026-08-03T21:03:21.339117+00:00 · methodology

0 comments
read the original abstract

Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-language models can build effective general-purpose audio encoders, nor a systematic understanding of how pretraining objectives behave across diverse tasks and scales. We identify three key barriers: limited scale of audio-text corpora, limited coverage of audio attributes in existing caption corpora, and lack of systematic exploration and evaluation. To fill this gap, we present the first principled empirical study of ALP. We first introduce CaptionStew, a 10.7M caption dataset aggregating open-source audio-text corpora across multiple domains and captioning focuses. We then conduct the first comprehensive evaluation comparing contrastive and captioning objectives for learning audio representation across speech, music, and environmental sound tasks. Our results not only demonstrate that ALP yields competitive, transferable representations, but reveal critical trade-offs: contrastive learning offers superior data efficiency, while captioning exhibits better scalability. Furthermore, we find that the benefits of supervised initialization often diminish at larger scales, challenging common practices. By grounding these claims in empirical evidence, we establish a viable pathway toward general-purpose audio representation learning, guiding future research.

Figures

Figures reproduced from arXiv: 2511.16757 by Dong Yu, Hao Zhang, Mingyue Huo, Wei-Cheng Tseng, Xuanru Zhou, Yiwen Shao.

Figure 1
Figure 1. Figure 1: Audio-language pretraining objective studied in this work: contrastive and captioning. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data scaling behavior of contrastive vs. captioning objectives across representative tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: t-SNE visualization of sentence embedding of captions grouped by source [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model diagram of Zipformer. A.3 ZIPFORMER MODEL In this work, we adopt the Zipformer-M architecture (Yao et al., 2024) as the audio encoder, cho￾sen for its memory efficiency on long sequences and strong performance across audio tasks. The architecture employs a U-Net-inspired design with six Transformer stages that process sequences at multiple temporal resolutions. The stages operate at progressively dec… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SoundscapeAgent: Agentic Soundscape Construction for Controllable Synthesis and Scalable Audio-Language Supervision

    cs.SD 2026-07 conditional novelty 6.0

    An agentic pipeline that plans, retrieves/generates, and deterministically renders multi-event soundscapes, and shows those structured outputs improve audio-language model reasoning over real-only data.

Reference graph

Works this paper leans on

52 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Youtube-8m: A large-scale video classification benchmark.arXiv preprint arXiv:1609.08675,

    Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark.arXiv preprint arXiv:1609.08675,

    Pith/arXiv arXiv

  2. [4]

    Common voice: A massively-multilingual speech corpus.arXiv preprint arXiv:1912.06670,

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus.arXiv preprint arXiv:1912.06670,

    Pith/arXiv arXiv 1912

  3. [7]

    Fusionaudio-1.2 m: Towards fine-grained audio captioning with multimodal contextual fusion.arXiv preprint arXiv:2506.01111,

    Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, and Benyou Wang. Fusionaudio-1.2 m: Towards fine-grained audio captioning with multimodal contextual fusion.arXiv preprint arXiv:2506.01111,

    Pith/arXiv arXiv

  4. [8]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pp. 1597–1607. PmLR, 2020b. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio techn...

    Pith/arXiv arXiv

  5. [14]

    Midashenglm: Efficient audio understanding with general audio captions.arXiv preprint arXiv:2508.03983,

    Heinrich Dinkel, Gang Li, Jizhong Liu, Jian Luan, Yadong Niu, Xingwei Sun, Tianzi Wang, Qiyang Xiao, Junbo Zhang, and Jiahao Zhou. Midashenglm: Efficient audio understanding with general audio captions.arXiv preprint arXiv:2508.03983,

    arXiv

  6. [15]

    Scaling rich style-prompted text- to-speech datasets.arXiv preprint arXiv:2503.04713,

    Anuj Diwan, Zhisheng Zheng, David Harwath, and Eunsol Choi. Scaling rich style-prompted text- to-speech datasets.arXiv preprint arXiv:2503.04713,

    arXiv

  7. [16]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736–740. IEEE,

    2020

  8. [17]

    Threshold independent evaluation of sound event detection scores

    Janek Ebbers, Reinhold Haeb-Umbach, and Romain Serizel. Threshold independent evaluation of sound event detection scores. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1021–1025. IEEE,

    2022

  9. [18]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  10. [20]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.arXiv preprint arXiv:2507.08128,

    Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao- Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.arXiv preprint arXiv:2507.08128,

    Pith/arXiv arXiv

  11. [21]

    Ast: Audio spectrogram transformer

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. InProc. Inter- speech 2021, pp. 571–575,

    2021

  12. [22]

    Ssast: Self-supervised audio spectrogram transformer

    12 Preprint Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10699–10709, 2022a. Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual...

    Pith/arXiv arXiv

  13. [23]

    The benefit of temporally-strong labels in audio event classification

    Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366–370. IEEE,

    2021

  14. [24]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 119–132,

    2019

  15. [25]

    Bootstrap- ping language-audio pre-training for music captioning

    Luca A Lanzend ¨orfer, Constantin Pinkl, Nathana ¨el Perraudin, and Roger Wattenhofer. Bootstrap- ping language-audio pre-training for music captioning. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2025

  16. [26]

    A diversity-promoting objective function for neural conversation models.arXiv preprint arXiv:1510.03055,

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models.arXiv preprint arXiv:1510.03055,

    Pith/arXiv arXiv

  17. [28]

    Music understanding llama: Advancing text-to-music generation with question answering and captioning

    Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding llama: Advancing text-to-music generation with question answering and captioning. InICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 286–290. IEEE,

    2024

  18. [29]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Pith/arXiv arXiv 1907

  19. [30]

    Mmar: A challenging benchmark for deep rea- soning in speech, audio, music, and their mix.arXiv preprint arXiv:2505.13032,

    Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. Mmar: A challenging benchmark for deep rea- soning in speech, audio, music, and their mix.arXiv preprint arXiv:2505.13032,

    Pith/arXiv arXiv

  20. [31]

    Expresso: A benchmark and analysis of discrete expressive speech resynthesis

    14 Preprint Tu Anh Nguyen, Wei-Ning Hsu, Antony D’Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. InProc. Interspeech 2023, pp. 4823–4827,

    2023

  21. [32]

    M2d-clap: Masked modeling duo meets clap for learning general- purpose audio-language representation.arXiv preprint arXiv:2406.02032,

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Masahiro Yasuda, Shunsuke Tsubaki, and Keisuke Imoto. M2d-clap: Masked modeling duo meets clap for learning general- purpose audio-language representation.arXiv preprint arXiv:2406.02032,

    Pith/arXiv arXiv

  22. [33]

    M2d2: Exploring general-purpose audio-language representations beyond clap

    Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Yasunori Ohishi, and Noboru Harada. M2d2: Exploring general-purpose audio-language representations beyond clap. arXiv preprint arXiv:2503.22104,

    arXiv

  23. [34]

    Representation learning with contrastive predic- tive coding.arXiv preprint arXiv:1807.03748,

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding.arXiv preprint arXiv:1807.03748,

    Pith/arXiv arXiv

  24. [35]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508,

    Pith/arXiv arXiv

  25. [36]

    Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084,

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084,

    Pith/arXiv arXiv 1908

  26. [37]

    Ears: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation

    Julius Richter, Yi-Chiao Wu, Steven Krenn, Simon Welker, Bunlong Lay, Shinji Watanabe, Alexan- der Richard, and Timo Gerkmann. Ears: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation. InProc. Interspeech 2024, pp. 4873–4877,

    2024

  27. [38]

    Jamendomaxcaps: A large scale music-caption dataset with imputed metadata.arXiv preprint arXiv:2502.07461,

    Abhinaba Roy, Renhang Liu, Tongyu Lu, and Dorien Herremans. Jamendomaxcaps: A large scale music-caption dataset with imputed metadata.arXiv preprint arXiv:2502.07461,

    Pith/arXiv arXiv

  28. [39]

    Audio-language models for audio-centric tasks: A survey.arXiv preprint arXiv:2501.15177,

    Yi Su, Jisheng Bai, Qisheng Xu, Kele Xu, and Yong Dou. Audio-language models for audio-centric tasks: A survey.arXiv preprint arXiv:2501.15177,

    Pith/arXiv arXiv

  29. [40]

    Hear: Holistic evaluation of audio representations

    Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Bj ¨orn W Schuller, Christian J Stein- metz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al. Hear: Holistic evaluation of audio representations. InNeurIPS 2021 Competitions and Demonstrations Track, pp. 125–145. PMLR,

    2021

  30. [41]

    Towards learning universal audio representations

    Luyu Wang, Pauline Luc, Yan Wu, Adria Recasens, Lucas Smaira, Andrew Brock, Andrew Jaegle, Jean-Baptiste Alayrac, Sander Dieleman, Joao Carreira, et al. Towards learning universal audio representations. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4593–4597. IEEE,

    2022

  31. [42]

    Wav2clip: Learning robust audio representations from clip

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567. IEEE,

    2022

  32. [43]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  33. [44]

    qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024a

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024a. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-lan...

    Pith/arXiv arXiv 1979

  34. [45]

    Speech self- supervised representation benchmarking: Are we doing it right? InInterspeech 2023, pp

    Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. Speech self- supervised representation benchmarking: Are we doing it right? InInterspeech 2023, pp. 2873– 2877,

    2023

  35. [46]

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer

    doi: 10.21437/Interspeech.2023-1087. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123–18133,

    doi:10.21437/interspeech.2023-1087 2023

  36. [47]

    Muq: Self-supervised music representation learning with mel residual vector quantization.arXiv preprint arXiv:2501.01108,

    Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Yi Luo, Wei Tan, and Xie Chen. Muq: Self-supervised music representation learning with mel residual vector quantization.arXiv preprint arXiv:2501.01108,

    Pith/arXiv arXiv

  37. [48]

    17 Preprint A APPENDIX A.1 LIMITATIONS While this work provides valuable empirical insights for audio-language pretraining, we acknowl- edge several important limitations that present opportunities for future research. Dataset Construction and Quality.CaptionStew aggregates captions from multiple sources with varying generation methodologies, including LL...

    2019

  38. [49]

    1.9M/1.9M4.0M/4.0M182k/182k general (environmental,human/animal sounds) AudioSet(Gemmeke et al., 2017)YouTube8M(Abu-El-Haija et al., 2016)VggSound(Chen et al., 2020a) LLM-generated, detailed,multi-sentence description three-stage pipeline:LALM attribute extraction→LLM captioning→CLAP-based filtering FusionAudio(Chen et al., 2025)1.2M/1.2M general (environ...

    2017

  39. [50]

    Human-annotated/LLM-augmented,speaking-style description crowdsourced /retrieval-basedmetadata imputation+ LALM captioning Table 6: Example caption sampled from each sourced dataset. Dataset Example Caption AudioCaps ”Distant traffic sounds followed by a car passing closely.” Clotho ”Something is being sanded or dragged, manipulated, scraped.” MusicCaps ”...

    2019

  40. [51]

    The architecture employs a U-Net-inspired design with six Transformer stages that process sequences at multiple temporal resolutions

    as the audio encoder, cho- sen for its memory efficiency on long sequences and strong performance across audio tasks. The architecture employs a U-Net-inspired design with six Transformer stages that process sequences at multiple temporal resolutions. The stages operate at progressively decreasing then increasing frame rates (50, 25, 12.5, 6.25, 12.5, and...

    2021

  41. [52]

    with fixed question template. Evaluation Dataset Task #samples #class train eval Metrics FSD-50k Multi-label audio event classification 37,168 / 10,231 200✓ ✓mAP VggSound Single-label audio event classification 183,730 / 15,446 309✓ ✓accuracy V oxCeleb2 Speaker identification 1,092,009 / 36,693 5,994✓ ✓accuracy CREMA-D Speech emotion recognition 6,030 / 7...

    2025

  42. [2015]

    Atst: Audio representation learning with teacher-student transformer.arXiv preprint arXiv:2204.12076,

    Xian Li and Xiaofei Li. Atst: Audio representation learning with teacher-student transformer.arXiv preprint arXiv:2204.12076,

    Pith/arXiv arXiv

  43. [2016]

    Musiclm: Generating music from text.arXiv preprint arXiv:2301.11325,

    Andrea Agostinelli, Timo I Denk, Zal ´an Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text.arXiv preprint arXiv:2301.11325,

    Pith/arXiv arXiv

  44. [2017]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6288–6313,

    2024

  45. [2018]

    Look, listen, and learn more: Design choices for deep audio embeddings

    Aurora Linh Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more: Design choices for deep audio embeddings. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852–3856. IEEE,

    2019

  46. [2019]

    Scaling up masked audio encoder learning for general audio classification

    Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang. Scaling up masked audio encoder learning for general audio classification. InProc. Interspeech 2024, pp. 547–551,

    2024

  47. [2020]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    11 Preprint Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,

    2019

  48. [2021]

    Vggsound: A large-scale audio- visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721–725. IEEE, 2020a. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts- at: A hierarchical token-se...

    2020

  49. [2022]

    Ecapa-tdnn: Emphasized chan- nel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143,

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized chan- nel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143,

    Pith/arXiv arXiv 2005

  50. [2023]

    Efficient supervised training of audio transformers for music representation learning

    Pablo Alonso-Jim´enez, Xavier Serra, and Dmitry Bogdanov. Efficient supervised training of audio transformers for music representation learning. InIsmir 2023 Hybrid Conference,

    2023

  51. [2024]

    V oxceleb2: Deep speaker recognition

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622,

    Pith/arXiv arXiv

  52. [2025]

    Openbeats: A fully open-source general-purpose audio encoder

    Shikhar Bharadwaj, Samuele Cornell, Kwanghee Choi, Satoru Fukayama, Hye-jin Shim, Soham Deshmukh, and Shinji Watanabe. Openbeats: A fully open-source general-purpose audio encoder. arXiv preprint arXiv:2507.14129,

    Pith/arXiv arXiv