Pith. sign in

REVIEW 4 major objections 5 minor 72 references

SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SLAP, a negative-free siamese pretraining method for text and audio, outperforms contrastive CLAP on music retrieval and zero-shot classification.

desk verdict SLAP is a legit new BYOL-style recipe for music-text with a controlled comparison and real modality-gap reduction, but headline gains rest on single runs and small margins; needs multi-seed evidence. read the letter →

arxiv 2506.17815 v1 pith:S6KC5OQC submitted 2025-06-21 cs.SD eess.AS

Julien Guinot , Alain Riou , Elio Quinton , György Fazekas This is my paper · ORCID
classification cs.SDeess.AS
keywords multimodalpretrainingmusic-textretrievalnegative-freelearningBYOLmodalitygapzero-shotclassificationself-supervisedmusicinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SLAP (Siamese Language-Audio Pretraining) replaces the contrastive loss at the heart of CLAP-style joint embedding training with a negative-free objective: for each music-caption pair, two EMA-updated target encoders provide slow-moving targets, and the context encoders predict those targets across both modalities using cosine losses. The paper argues this yields embeddings that beat a same-architecture CLAP baseline on text-music retrieval and zero-shot classification, match or exceed it on downstream genre, tag, and instrument probes, and cut the modality gap to near zero. If true, joint text-music spaces no longer need large batches of negative samples, making large-scale training cheaper and better suited to many-to-many music-caption relationships.

What carries the argument

The central machinery is an asymmetric siamese network with two branches per modality: a context encoder $E$ that gets gradients and an EMA target encoder $\bar{E}$ updated as $\tau\bar{E} + (1-\tau)E$, plus a predictor MLP on each context branch. The predictors produce $q_A$ and $q_T$, each of which is trained to match the EMA targets of both audio and text via cosine losses. The stop-gradient on target encoders and the asymmetry-breaking predictor are what prevent collapse in principle, while the intramodal terms $L_A$ and $L_T$ keep the two modalities from drifting apart. This setup lets the objective be computed without any cross-sample negative pairs, so the loss does not need a $B\times B$ similarity matrix and permits gradient accumulation.

What would settle it

Train SLAP with $\lambda=0.5$ on a public music-caption dataset (for example MusicCaps captions paired with their audio) instead of PrivateCaps and measure audio-to-text retrieval and the audio/text linear separability; if retrieval collapses to chance or the separability becomes as large as CLAP's, the collapse-avoidance claim fails. A second check: sweep the EMA rate $\tau$ from 0.9 to 0.999 and observe whether retrieval degrades sharply outside 0.95, which would show the claimed robustness does not generalise.

Watch

Extended reading notes

Core claim

On its own terms, SLAP's discovery is that the multimodal contrastive loss is not necessary for strong text-music joint embeddings. Starting from the same HTS-AT audio encoder and RoBERTa text encoder used in the LAION-CLAP model, and training on the same PrivateCaps pairs, the paper's loss $L = \lambda(L_{A\to T}+L_{T\to A}) + (1-\lambda)(L_A+L_T)$ — where each $L$ is a cosine distance between a predictor output and an EMA target — outperforms the reproduced CLAP on retrieval recall and normalized rank, raises zero-shot accuracy on GTZAN from 51.7 to 58.3, and lowers both the centroid distance and linear separability between audio and text manifolds. The authors claim that the intramodal losses are necessary to prevent collapse, and that only a balanced weight $\lambda\approx 0.5$ removes the modality gap while preserving retrieval.

Load-bearing premise

The training dynamics do not collapse: the slowly-updated target encoders and the predictor networks must keep all branches producing informative, non-identical representations, and the mixing weight $\lambda$ must stay close to 0.5 so the within-modality and between-modality losses stay balanced; the paper verifies this only on one private dataset and a narrow $\lambda$ range.

Editorial extensions

If this is right

  • Text-music retrieval improves with no negative samples: SLAP beats CLAP on Recall@1/5/10 and normalized rank on Song Describer and MusicCaps for both retrieval directions.
  • Zero-shot music classification and tagging improve: best-of-four prompts give 58.3 percent accuracy on GTZAN, 31.5 percent mAP on MTAT, and 70.5 percent mAR on OpenMic, all above the CLAP baseline.
  • Frozen SLAP audio embeddings transfer to downstream probing: they beat CLAP and MusCALL on genre, tag, and instrument tasks and reach the best mAP on MagnaTagATune among the compared models.
  • The modality gap nearly disappears with a balanced loss weight $\lambda=0.5$, as measured by lower audio/text centroid distance and lower linear separability.
  • Because the loss needs no negatives, gradient accumulation makes effective batch sizes much larger on a single GPU; the paper shows retrieval stays stable as batch size grows from 64 to 768.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the collapse-avoidance property holds across datasets, SLAP-style losses could replace contrastive heads in other paired modalities (image-text, video-text) where many-to-many pairing and long-tailed data make negative sampling awkward.
  • The near-zero modality gap suggests SLAP embeddings may be directly usable as conditioning latents for text-to-music generation, where disjoint manifolds have been identified as an obstacle; this is a testable next step the paper does not run.
  • The narrow safe range for $\lambda$ (about 0.2 to 0.7, and exactly 0.5 for minimal gap) implies the method needs a dataset-specific balancing of the four losses; a public dataset replication would show whether the optimum migrates with caption noise or batch size.
  • The claimed robustness to batch size implies that very large effective batches could be reached by gradient accumulation alone, which would let researchers train on today's largest caption corpora without multi-GPU memory walls.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SLAP, a negative-free multimodal pretraining framework that adapts the BYOL paradigm to music-text learning. Audio and text encoders are trained with an EMA-updated target branch and predictors, optimizing a weighted sum of intermodal and intramodal cosine losses (Eq. 5). The authors evaluate SLAP against a reproduced CLAP baseline with identical architecture, training data, and budget on text-music retrieval (MusicCaps, Song Describer), zero-shot classification (GTZAN, MTAT, OpenMic), and downstream probing, and additionally report a reduced modality gap and improved robustness to batch-size variation. The central claim is that SLAP outperforms or matches CLAP while avoiding large-batch negative-sample requirements.

Significance. If the reported results hold, the contribution is valuable: it offers a non-contrastive alternative to CLAP-style training for music-language embeddings, with a demonstrably smaller modality gap and the practical benefit of gradient accumulation. The experimental design is a strength: SLAP and the reproduced CLAP baseline share the same architecture, training data, and computational budget, and the public code release supports reproducibility. The reduced modality gap is quantified with both centroid distance and linear separability, and the batch-size study directly addresses the scalability motivation. The main weakness is statistical: the headline retrieval comparisons rest on single runs with small absolute margins, and the loss-weight selection uses the same test sets that later serve as evaluation.

major comments (4)
  1. [Section 5.1, Table 2] All retrieval results are single training runs with no error bars, confidence intervals, or significance tests. The SLAP-vs-CLAP margins are small (e.g., 3.1 vs 2.8 R@1 on MusicCaps A→T and 5.7 vs 5.3 on Song Describer A→T), and with roughly 5.5k and 1k test queries these gaps correspond to only a handful of correctly ranked items, which could easily be within seed-to-seed variation. Please report multiple seeds with mean and standard deviation, or otherwise provide statistical support for the claim that SLAP consistently outperforms CLAP on retrieval.
  2. [Section 5.6, Figure 5] The balancing weight λ is tuned on MusicCaps and Song Describer, which are the same datasets used for the headline retrieval results in Table 2. This creates an optimism bias in the reported advantages, because the model is effectively selected on the evaluation sets. Please tune λ on a held-out validation split, or report results for a λ chosen without access to the test data, and indicate whether the Table 2 conclusions remain unchanged.
  3. [Section 5.6, Figure 5] The viability of the method depends on the EMA mechanism and predictors preventing representation collapse, but the paper reports that values of λ outside [0.2, 0.7] lead to near-systematic collapse and provides no seed variance or quantitative collapse diagnostic for the stable regime. Since a single collapsed run would invalidate the learned embeddings, please report multiple seeds, a collapse measure on validation data, and discuss the narrow stable range in relation to the claimed hyperparameter robustness.
  4. [Section 5.5, Figure 4] The statement that gradient accumulation in SLAP is empirically confirmed to be equivalent to true batch scaling is supported only by a single set of curves on PrivateCaps without error bars or a direct side-by-side comparison with non-accumulated training at the same effective batch size. Please provide repeated runs or explicitly label this as an informal observation, and clarify whether the equivalence claim refers to optimization dynamics or final retrieval performance.
minor comments (5)
  1. [Title and running text] The title in the full-text version contains a spacing typo, 'NEGA TIVE', which should be corrected to 'NEGATIVE'.
  2. [Section 3] The text encoder definition says 'maps Ntokens to a tex latent space'; 'tex' should be 'text'.
  3. [Abstract and Section 4.3] The abstract claims that the formulation 'unlocks large-scale training on a single GPU through gradient accumulation', but Section 4.3 states that all models were trained on 6 A100 GPUs. Please clarify whether single-GPU training was actually run or is only a potential benefit.
  4. [Introduction and Section 5.6] The paper describes the approach as 'hyperparameter-robust', but Figure 5 shows collapse outside λ ∈ [0.2, 0.7]; please qualify this claim to the stable region or provide additional evidence of robustness.
  5. [Table 5 and Eq. (6)] The definition of mAR uses R_n@k without explicitly stating whether it is recall at k averaged over tags and samples in the same way as the standard multilabel retrieval metric; a one-sentence clarification would improve reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

The reduced modality gap is the training objective itself, and the λ hyperparameter is tuned on the same retrieval test sets reported in Table 2; the main retrieval/probing outperformance claims rest on independent external evaluations.

  1. self definitional [Section 3, Eq. (3)-(5) and Section 5.4]
    "The model is trained by minimizing intermodal losses, defined as the cosine distance between the queries q and targets ¯z from different modalities: ... These observations confirm the modality gap is larger, both in terms of distance and linear separability, for CLAP than for SLAP."

    The intermodal loss in Eq. (3) is literally the cosine distance between audio and text embeddings, and the Section 5.4 modality-gap metrics are centroid distance and linear separability between those same audio and text embedding manifolds. Minimizing Eq. (3) is therefore directly minimizing the measured gap; reporting a 'quantifiably reduced modality gap' for SLAP is a restatement of the training objective, not an independent empirical discovery. The held-out measurements show generalization, but the gap reduction itself is imposed by construction.

  2. fitted input called prediction [Section 5.6, Figure 5 and Table 2]
    "Tuning of intramodality loss balancing weight λ for the SLAP objective, measuring both T→A and A→T retrieval performance and L A/T . ... Values of λ outside [0.2, 0.7] lead to almost systematic collapse due to the lack of meaningful constraints between or within modalities."

    Figure 5 sweeps λ on MusicCaps and Song Describer, and Section 5.6 concludes that a balanced λ (0.5) is needed. Table 2 then reports SLAP's retrieval results on those exact same MusicCaps and Song Describer test sets. The headline retrieval advantage is therefore partly the result of selecting the balancing weight on the evaluation data; the reported numbers are not an unbiased prediction but a test-set-selected outcome.

full rationale

SLAP's core outperformance claims (retrieval, zero-shot, probing) are evaluated against a reproduced CLAP baseline on the same private training data and on external public test sets (MusicCaps, Song Describer, GTZAN, MTAT, OpenMic). Those comparisons are not circular: the CLAP baseline is trained with the same architecture and data, and the test sets are external. However, two components of the paper's contribution list are circular or test-set-fitted. First, the 'quantifiably reduced modality gap' is the direct objective of Eq. (3): the intermodal loss is defined as the cosine distance between audio and text embeddings, and Section 5.4 measures that same distance/separability. This is a self-definitional result rather than an emergent property. Second, the λ balancing weight is tuned on MusicCaps and Song Describer in Figure 5, and the same datasets are used for the headline Table 2 retrieval results, so the reported advantage includes test-set selection bias. The single-run, small-margin differences (e.g., 0.2–0.4 R@1) further weaken the outperformance claim, but that is a statistical robustness concern, not circularity. The use of PrivateCaps from the authors' prior work is a reproducibility limitation, not a circular derivation, since both methods are trained on it. Overall, the central 'SLAP beats CLAP' claim retains independent content, so this is partial circularity rather than a fully forced result.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims depend on a handful of training hyperparameters (loss weight lambda, EMA rate tau, predictor width, accumulation base batch) and on domain assumptions about collapse avoidance, the sufficiency of cosine-distance alignment, and the representativeness of the private training set. No fundamentally new entities are introduced.

free parameters (4)
  • Loss weight lambda = 0.5 (balanced; implied, not explicitly stated for main runs)
    Balances intermodal and intramodal loss terms in Equation (5); Section 5.6 shows only lambda in [0.2, 0.7] avoids collapse and that only lambda=0.5 produces the reduced modality gap.
  • EMA update rate tau = 0.95
    Target encoder update rate in Equation (1); chosen empirically over the 0.996 used in BYOL-Audio [11] based on retrieval performance, Section 4.3.
  • Predictor width = 4096 (one hidden layer)
    MLP hidden width for predictors P_A and P_T; stated as a default with tuned-study left to future work, Section 4.2.
  • Gradient accumulation base batch = 128 (accumulated to effective 768)
    SLAP uses a base batch of 128 with accumulation to reach the target batch size; the equivalence with true batch scaling is asserted empirically in Section 5.5.
assumptions (4)
  • domain assumption EMA update with a predictor prevents representation collapse in the multimodal setting exactly as it does in unimodal BYOL.
    Section 3 and Section 5.6 rely on [10, 38] for collapse avoidance; the paper shows empirically that within lambda in [0.2, 0.7] collapse is avoided, but no proof is given for the cross-modal case.
  • domain assumption Cosine distance between predictor outputs and EMA target outputs across modalities is a sufficient training signal to learn aligned audio-text embeddings.
    Equations (3) to (5) define the alignment objective; the paper verifies this empirically on PrivateCaps, but there is no theoretical argument for sufficiency.
  • domain assumption The private PrivateCaps training set (260k professionally captioned full tracks) provides a representative distribution for music-text alignment.
    Section 4.1; the central results depend on this data, whose characteristics and collection protocol are not publicly available.
  • domain assumption Cross-modal retrieval and zero-shot classification using inner products of these embeddings are valid proxies for representation quality.
    Sections 5.1 to 5.3 evaluate the model under the assumption that cosine similarity in the joint space corresponds to semantic relevance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding." pith.science (2026). https://pith.science/paper/S6KC5OQC

@misc{pith2026250617815,
  author       = {Pith},
  title        = {Pith review of: SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6KC5OQC}},
  note         = {Machine review of arXiv:2506.17815}
}
read the original abstract

Joint embedding spaces have significantly advanced music understanding and generation by linking text and audio through multimodal contrastive learning. However, these approaches face large memory requirement limitations due to relying on large batch sizes to effectively utilize negative samples. Further, multimodal joint embedding spaces suffer from a modality gap wherein embeddings from different modalities lie in different manifolds of the embedding space. To address these challenges, we propose Siamese Language-Audio Pretraining (SLAP), a novel multimodal pretraining framework that allows learning powerful representations without negative samples. SLAP adapts the Bootstrap Your Own Latent (BYOL) paradigm for multimodal audio-text training, promoting scalability in training multimodal embedding spaces. We illustrate the ability of our model to learn meaningful relationships between music and text -- specifically, we show that SLAP outperforms CLAP on tasks such as text-music retrieval and zero-shot classification. We also observe competitive downstream performance on several MIR tasks, including with larger or supervised models (genre and instrument classification, auto-tagging). Additionally, our approach has attractive properties, such as a quantifiably reduced modality gap and improved robustness to batch size variations on retrieval performance. Finally, its novel formulation unlocks large-scale training on a single GPU through gradient accumulation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 57 canonical work pages

  1. [1]

    SLAP: Siamese Language-Audio Pretraining without negative samples for Music Understanding

    INTRODUCTION Joint embedding spaces for text and audio have been foun- dational in recent developments in music understanding and generation. Such spaces are typically learned via Mul- timodal Contrastive Learning (MCL), which optimizes a pair of encoders for maximal similarity between positive pairs, while minimizing similarity for negative pairs [1, 2]....

    work page 2025

  2. [2]

    We introduce a scalable, hyperparameter-robust ap- proach to language-audio pretraining which does not require negative pairs to learn strong representations

    work page

  3. [3]

    We outperform comparable contrastive models on retrieval and downstream probing

    work page

  4. [4]

    We show that our approach significantly decreases the modality gap between audio and text embed- dings compared to contrastive approaches

    work page

  5. [5]

    SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding

    Our approach enables larger batch sizes via gradi- ent accumulation, which was previously inaccessi- ble due to the formulation of the contrastive loss. To facilitate further research in this direction, we make our code available. 1 1 https://github.com/Pliploop/SLAP arXiv:2506.17815v1 [cs.SD] 21 Jun 2025

    work page Pith review arXiv 2025

  6. [6]

    BACKGROUND 2.1 Multimodal Contrastive Learning In contrastive learning, models learn representations by maximizing the similarity between twoviewsof an in- put while minimizing its similarity withnegative sam- ples, typically by optimizing an InfoNCE loss [12]. While these views are typically crafted by randomly applying transforms to the input data point...

    work page

  7. [7]

    A blaring metal track with stompy kicks and distorted chuggy guitar

    SIAMESE LANGUAGE-AUDIO PRETRAINING The training pipeline of SLAP is depicted in Figure 1. Consider an audio encoderE A mapping samplesx A of lengthT A from the audio space to a latent audio repre- sentationz a so thatE A :x A ∈R TA 7→z A ∈R d and a text encoder mappingNtokens to a tex latent space ET :x t∈N N7→z t∈R d. Audio and text encoders project inpu...

    work page

  8. [8]

    We train SLAP on an internal private dataset of 260,000 pairs of full-length production-quality music tracks and professionally annotated captions (PrivateCaps [16])

    EXPERIMENTAL SETUP 4.1 Datasets The list of datasets used in this work is reported in Table 1. We train SLAP on an internal private dataset of 260,000 pairs of full-length production-quality music tracks and professionally annotated captions (PrivateCaps [16]). Our retrieval testing datasets include two music-caption pair datasets. Specifically, we use Mu...

    work page

Show all 72 references
  1. [9]

    {}”, “{} music

    RESULTS 5.1 Multimodal retrieval We perform Audio to Text (A→T) and Text to Audio (T→A) retrieval across the test datasets described in Sec- tion 4.1. We use predictionsqas key and query for SLAP models (see Section 5.1.1) and projectionszfor CLAP models. We report Recall @(1,...

  2. [10]

    SLAP out- performs contrastive models on tasks including text-music retrieval, downstream probing, and zero-shot music un- derstanding

    CONCLUSION AND FUTURE WORK We propose a novel approach to training multimodal joint embedding spaces to align music and text representation: Siamese Language Audio Pretraining (SLAP). SLAP out- performs contrastive models on tasks including text-music retrieval, downstream pro...

  3. [11]

    ACKNOWLEDGEMENT This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1) and Universal Music Group

  4. [12]

    Learn- ing transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacyet al., “Learn- ing transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  5. [13]

    Clap learning audio concepts from natural language supervi- sion,

    B. Elizalde, S. Deshmukh, M. Al Ismailet al., “Clap learning audio concepts from natural language supervi- sion,” inICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  6. [14]

    Collap: Contrastive long-form language-audio pretraining with musical temporal structure augmentation,

    J. Wu, W. Li, Z. Novacket al., “Collap: Contrastive long-form language-audio pretraining with musical temporal structure augmentation,” inICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  7. [15]

    Aligned contrastive learning for text-to-music retrieval,

    T. Komatsu, H. Munakata, T. Hasumiet al., “Aligned contrastive learning for text-to-music retrieval,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  8. [16]

    T-clap: Temporal- enhanced contrastive language-audio pretraining,

    Y . Yuan, Z. Chen, X. Liuet al., “T-clap: Temporal- enhanced contrastive language-audio pretraining,” in 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1–6

    2024

  9. [17]

    Augment, drop & swap: Improving diversity in llm captions for efficient music-text representation learning,

    I. Manco, J. Salamon, and O. Nieto, “Augment, drop & swap: Improving diversity in llm captions for efficient music-text representation learning,” inProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024

    2024

  10. [18]

    Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap,

    A. Fahim, A. Murphy, and A. Fyshe, “Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap,” may 2024. [Online]. Available: http://arxiv.org/abs/2405.18570

    2024 arXiv

  11. [19]

    Mind the gap: Understanding the modality gap in multi-modal con- trastive representation learning,

    V . W. Liang, Y . Zhang, Y . Kwonet al., “Mind the gap: Understanding the modality gap in multi-modal con- trastive representation learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 17 612– 17 625, 2022

    2022

  12. [20]

    Combined scaling for zero-shot transfer learning,

    H. Pham, Z. Dai, G. Ghiasiet al., “Combined scaling for zero-shot transfer learning,”Neurocomputing, vol. 555, p. 126658, 2023

    2023

  13. [21]

    Bootstrap your own latent-a new approach to self-supervised learn- ing,

    J.-B. Grill, F. Strub, F. Altchéet al., “Bootstrap your own latent-a new approach to self-supervised learn- ing,”Advances in neural information processing sys- tems, vol. 33, pp. 21 271–21 284, 2020

    2020

  14. [22]

    Byol for au- dio: Self-supervised learning for general-purpose au- dio representation,

    D. Niizumi, D. Takeuchi, Y . Ohishiet al., “Byol for au- dio: Self-supervised learning for general-purpose au- dio representation,” in2021 International Joint Con- ference on Neural Networks (IJCNN). IEEE, 2021, pp. 1–8

    2021

  15. [23]

    A simple framework for contrastive learning of visual represen- tations,

    T. Chen, S. Kornblith, M. Norouziet al., “A simple framework for contrastive learning of visual represen- tations,” inInternational conference on machine learn- ing. PMLR, 2020, pp. 1597–1607

    2020

  16. [24]

    Contrastive learning of musical representations,

    J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” inProceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR), 2021

    2021

  17. [25]

    V ATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text,

    H. Akbari, L. Yuan, R. Qian, W. H. Chuang, S. F. Chang, Y . Cui, and B. Gong, “V ATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text,” inAdvances in Neural Information Processing Systems, vol. 29. Neural information processing systems found...

    2021 arXiv

  18. [26]

    Look, Listen and Learn,

    R. Arandjelovic and A. Zisserman, “Look, Listen and Learn,” inProceedings of the IEEE International Conference on Computer Vision, vol. 2017-Octob. Institute of Electrical and Electronics Engineers Inc., may 2017, pp. 609–617. [Online]. Available: https://arxiv.org/abs/1705.08168v2

    2017 arXiv

  19. [27]

    Contrastive audio-language learning for music,

    I. Manco, E. Benetos, E. Quintonet al., “Contrastive audio-language learning for music,” inProceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, P. Rao, H. A. Murthy, A. Srini- vasamurthy, R....

    2022

  20. [28]

    Mulan: A joint em- bedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Leeet al., “Mulan: A joint em- bedding of music audio and natural language,” inPro- ceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Ben- galuru, India, December 4-8, 2022, P. Rao, H. A. Murthy, A. Srini...

    2022

  21. [29]

    Cacophony: An improved contrastive audio-text model,

    G. Zhu, J. Darefsky, and Z. Duan, “Cacophony: An improved contrastive audio-text model,”IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024

    2024

  22. [30]

    Audioclip: Ex- tending clip to image, text and audio,

    A. Guzhov, F. Raue, J. Heeset al., “Audioclip: Ex- tending clip to image, text and audio,” inICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 976–980

    2022

  23. [31]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liuet al., “Imagebind: One embedding space to bind them all,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15 180–15 190

    2023

  24. [32]

    Gramian Multimodal Representation Learn- ing and Alignment,

    G. Cicchetti, E. Grassucci, L. Sigillo, and D. Com- miniello, “Gramian Multimodal Representation Learn- ing and Alignment,” dec 2024. [Online]. Available: http://arxiv.org/abs/2412.11959

    2024 arXiv

  25. [33]

    Hierarchi- cal text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nicholet al., “Hierarchi- cal text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, vol. 1, no. 2, p. 3, 2022

    2022 arXiv

  26. [34]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Lucet al., “Flamingo: a visual language model for few-shot learning,”Ad- vances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  27. [35]

    Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities,

    Z. Kong, A. Goel, R. Badlaniet al., “Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities,” inProceedings of Machine Learning Research, vol. 235. ML Research Press, feb 2024, pp. 25 125–25 148. [Online]. Available: https://arxiv.org/abs/24...

    2024 arXiv

  28. [36]

    Reclap: Improving zero shot audio classification by describ- ing sounds,

    S. Ghosh, S. Kumar, C. K. R. Evuruet al., “Reclap: Improving zero shot audio classification by describ- ing sounds,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  29. [37]

    Drcap: Decoding clap la- tents with retrieval-augmented generation for zero-shot audio captioning,

    X. Li, W. Chen, Z. Maet al., “Drcap: Decoding clap la- tents with retrieval-augmented generation for zero-shot audio captioning,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  30. [38]

    Recap: Retrieval-augmented audio captioning,

    S. Ghosh, S. Kumar, C. K. R. Evuruet al., “Recap: Retrieval-augmented audio captioning,” inICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1161–1165

    2024

  31. [39]

    Fast timing- conditioned latent audio diffusion,

    Z. Evans, C. Carr, J. Tayloret al., “Fast timing- conditioned latent audio diffusion,” inProceedings of the 41st International Conference on Machine Learn- ing, 2024, pp. 12 652–12 665

    2024

  32. [40]

    Long-form mu- sic generation with latent diffusion,

    Z. Evans, J. D. Parker, C. Carret al., “Long-form mu- sic generation with latent diffusion,” inProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024

    2024

  33. [41]

    Diff-a-riff: Musical accompaniment co-creation via latent diffu- sion models,

    J. Nistal, M. Pasini, C. Aouameuret al., “Diff-a-riff: Musical accompaniment co-creation via latent diffu- sion models,” inISMIR, 2024, 2024

    2024

  34. [42]

    AudioLDM: Text-to- audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuanet al., “AudioLDM: Text-to- audio generation with latent diffusion models,” inProc. ICML, 2023

    2023

  35. [43]

    Mu- sicLM: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsoset al., “Mu- sicLM: Generating music from text,”arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  36. [44]

    Sigmoid Loss for Language Image Pre-Training,

    X. Zhai, B. Mustafa, A. Kolesnikovet al., “Sigmoid Loss for Language Image Pre-Training,” inProceed- ings of the IEEE International Conference on Com- puter Vision. Institute of Electrical and Electronics Engineers Inc., mar 2023, pp. 11 941–11 952. [Online]. Available: https:...

    2023 arXiv

  37. [45]

    The Hidden Uniform Cluster Prior in Self-Supervised Learning,

    M. Assran, R. Balestriero, Q. Duvalet al., “The Hidden Uniform Cluster Prior in Self-Supervised Learning,” in11th International Conference on Learning Representations, ICLR 2023, oct 2023. [Online]. Available: http://arxiv.org/abs/2210.07277

    2023 arXiv

  38. [46]

    Understand- ing the Modality Gap in CLIP,

    P. Y . Shi, M. Welle, M. Björkmanet al., “Understand- ing the Modality Gap in CLIP,” inInternational Con- ference on Learning Reprsentations, no. 2023, 2023

    2023

  39. [47]

    Exploring simple siamese repre- sentation learning,

    X. Chen and K. He, “Exploring simple siamese repre- sentation learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2021, pp. 15 750–15 758

    2021

  40. [48]

    Self-Supervised Learning from Images with a Joint-Embedding Predic- tive Architecture,

    M. Assran, Q. Duval, I. Misraet al., “Self-Supervised Learning from Images with a Joint-Embedding Predic- tive Architecture,” in2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), jun 2023, pp. 15 619–15 629, iSSN: 2575-7075

    2023

  41. [49]

    Understanding Self- Supervised Learning Dynamics without Contrastive Pairs,

    Y . Tian, X. Chen, and S. Ganguli, “Understanding Self- Supervised Learning Dynamics without Contrastive Pairs,”Proceedings of Machine Learning Research, vol. 139, pp. 10 268–10 278, feb 2021. [Online]. Available: http://arxiv.org/abs/2102.06810

    2021 arXiv

  42. [50]

    DINOv2: Learning Robust Visual Features without Supervision,

    M. Oquab, T. Darcet, T. Moutakanniet al., “DINOv2: Learning Robust Visual Features without Supervision,” apr 2023. [Online]. Available: http://arxiv.org/abs/ 2304.07193

    2023 arXiv

  43. [51]

    data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,

    A. Baevski, W.-N. Hsu, Q. Xuet al., “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,” inProceedings of Ma- chine Learning Research, Baltimore, MD, USA, 2022

    2022

  44. [52]

    ATST: Audio Representa- tion Learning with Teacher-Student Transformer,

    X. Li and X. Li, “ATST: Audio Representa- tion Learning with Teacher-Student Transformer,” inProceedings of the Annual Conference of the International Speech Communication Asso- ciation, INTERSPEECH, vol. 2022-Septe. In- ternational Speech Communication Association, apr 2022, ...

    2022 doi

  45. [53]

    Masked Modeling Duo: Learning Representations by Encour- aging Both Networks to Model the Input,

    D. Niizumi, D. Takeuchi, Y . Ohishiet al., “Masked Modeling Duo: Learning Representations by Encour- aging Both Networks to Model the Input,” inICASSP 2023 - 2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). Rhodes Island, Greece: IEEE, ...

    2023

  46. [54]

    Masked latent prediction and classification for self-supervised audio representation learning,

    A. Quelennec, P. Chouteau, G. Peeterset al., “Masked latent prediction and classification for self-supervised audio representation learning,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  47. [55]

    The song describer dataset: a corpus of audio captions for music-and- language evaluation,

    I. Manco, B. Weck, S. Dohet al., “The song describer dataset: a corpus of audio captions for music-and- language evaluation,” NeurIPS Machine Learning for Audio Workshop, 2023

    2023

  48. [56]

    Musical genre classification of audio signals,

    G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,”IEEE Transac- tions on Speech and Audio Processing, vol. 10, no. 5, pp. 293–302, 2002. [Online]. Available: https://doi.org/10.1109/TSA.2002.800560

    2002

  49. [57]

    Evaluation of Algorithms Using Games: The Case of Music Tagging,

    E. Law, K. West, M. I. Mandelet al., “Evaluation of Algorithms Using Games: The Case of Music Tagging,” inProceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center , Kobe, Japan, October 26-30, 20...

    2009

  50. [58]

    Openmic- 2018: An open data-set for multiple instrument recog- nition

    E. Humphrey, S. Durand, and B. McFee, “Openmic- 2018: An open data-set for multiple instrument recog- nition.” inISMIR, 2018, pp. 438–444

    2018

  51. [59]

    Au- dio set: An ontology and human-labeled dataset for au- dio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedmanet al., “Au- dio set: An ontology and human-labeled dataset for au- dio events,” inProc. IEEE ICASSP 2017, New Orleans, LA, 2017

    2017

  52. [60]

    Large-scale con- trastive language-audio pretraining with feature fu- sion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhanget al., “Large-scale con- trastive language-audio pretraining with feature fu- sion and keyword-to-caption augmentation,” inProc. ICASSP. IEEE, 2023, pp. 1–5

    2023

  53. [61]

    Hts-at: A hierarchical token-semantic audio transformer for sound classifica- tion and detection,

    K. Chen, X. Du, B. Zhuet al., “Hts-at: A hierarchical token-semantic audio transformer for sound classifica- tion and detection,” inICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 646–650

    2022

  54. [62]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyalet al., “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  55. [63]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhanget al., “Specaugment: A simple data augmentation method for automatic speech recognition,”Interspeech 2019, p. 2613, 2019

    2019

  56. [64]

    Samplematch: Drum sample retrieval by musical context,

    S. Lattner, “Samplematch: Drum sample retrieval by musical context,” inProceedings of the 23rd International Society for Music Information Re- trieval Conference, {ISMIR} 2022, Bengaluru, India, December 4-8, 2022, 8 2022, pp. 781–788. [On- line]. Available: http://arxiv.org/a...

    2022 arXiv

  57. [65]

    Ef- ficient training of audio transformers with patchout,

    K. Koutini, J. Schlüter, H. Eghbal-zadehet al., “Ef- ficient training of audio transformers with patchout,” Interspeech 2022, 2022

    2022

  58. [66]

    Codified au- dio language modeling learns useful representations for music information retrieval,

    R. Castellon, C. Donahue, and P. Liang, “Codified au- dio language modeling learns useful representations for music information retrieval,”International Society for Music Information Retrieval (ISMIR), 2021

    2021

  59. [67]

    Beats: audio pre- training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wanget al., “Beats: audio pre- training with acoustic tokenizers,” inProceedings of the 40th International Conference on Machine Learn- ing, 2023, pp. 5178–5193

    2023

  60. [68]

    Improving mu- sical accompaniment co-creation via diffusion trans- formers,

    J. Nistal, M. Pasini, and S. Lattner, “Improving mu- sical accompaniment co-creation via diffusion trans- formers,” inAudio Imagination: NeurIPS 2024 Work- shop AI-Driven Speech, Music, and Sound Generation, 2024

    2024

  61. [69]

    On the Language Encoder of Contrastive Cross-modal Models,

    M. Zhao, J. Ono, Z. Zhonget al., “On the Language Encoder of Contrastive Cross-modal Models,” oct

  62. [71]

    Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities,

    S. Ghosh, Z. Kong, S. Kumaret al., “Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities,” mar 2025. [Online]. Available: http: //arxiv.org/abs/2503.03983

    2025 arXiv

  63. [72]

    MaskCLIP: Masked Self- Distillation Advances Contrastive Language-Image Pretraining,

    X. Dong, J. Bao, Y . Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen, F. Wen, and N. Yu, “MaskCLIP: Masked Self- Distillation Advances Contrastive Language-Image Pretraining,” pp. 10 995–11 005, aug 2023. [Online]. Available: https://arxiv.org/abs/2208.1...

    2023 arXiv

  64. [2023]

    Available: http://arxiv.org/abs/2310

    [Online]. Available: http://arxiv.org/abs/2310. 13267

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.