Pith. sign in

REVIEW 3 major objections 7 minor 55 references

Emergent musical properties of a transformer under contrastive self-supervised learning

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that training a transformer with a contrastive loss on its class token alone still yields locally informative musical representations in its sequence tokens, approaching much larger masked-modeling models on beat…

desk verdict A small class-token-only contrastive ViT does develop usable local musical representations, but the strongest direct evidence (attentional onset detection) rests on a single hand-picked head with no reported distribution. read the letter →

arxiv 2506.23873 v1 pith:ZQ7YVWPK submitted 2025-06-30 cs.SD cs.IRcs.LGeess.AS

Yuexuan Kong , Gabriel Meseguer-Brocal , Vincent Lostanlen , Mathieu Lagrange , Romain Hennequin This is my paper
classification cs.SDcs.IRcs.LGeess.AS
keywords contrastiveself-supervisedlearningmusicinformationretrievalvisiontransformersequencetokensemergentmusicalpropertiesonsetdetectionbeattrackingchordestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the common assumption in music information retrieval that contrastive self-supervised learning only supports whole-song tasks and that frame-level tasks such as beat tracking or chord estimation require masked modeling. The authors train a lightweight vision transformer with one-dimensional time-frequency patches using only a contrastive loss applied to the class token, then probe the frozen representations. They find that the sequence tokens, which are never directly supervised, carry enough harmonic and rhythmic information to come close to a much larger masked-modeling model on beat tracking and chord estimation. Averaging the class and sequence tokens improves global-task performance over the class token alone, and attention maps from a deep layer align with note onsets. A reader should care because, if true, simple contrastive training can produce local musical representations without the cost of masked modeling.

What carries the argument

The load-bearing object is a one-dimensional vision transformer (ViT-1D) whose input tokens are individual mel-spectrogram frames. A class token, formed from learnable parameters plus the average of the sequence tokens, is prepended and the normalized temperature-scaled cross-entropy (NT-Xent) loss is applied only to that class token. Because the class token shares weights with the sequence tokens in the transformer's attention and MLP blocks, the contrastive gradient indirectly shapes every token. The analyses then convert the model's internal structure into musical evidence: layer-wise attention maps are averaged per column to make an onset pseudo-activation, and self-similarity matrices on the token sequence reveal which musical dimensions each layer encodes.

What would settle it

Apply the paper's peak-picking procedure to every attention head across all 12 layers and several training seeds: if most trained heads yield onset F-scores near the random-initialization value of 0.501 rather than the reported 0.877, the claimed emergence of onset alignment is selective rather than systemic.

Watch

Extended reading notes

Core claim

The paper's central discovery is that time-invariance imposed on the class token does not prevent temporal structure from appearing elsewhere in the model. On beat tracking the sequence tokens reach F-score 0.723 against 0.794 for a masked-modeling baseline, and on chord estimation they reach accuracy 0.319 against 0.322, despite having roughly one seventeenth of the parameters. The class token itself remains time-invariant, but the sequence tokens become locally informative through the shared weights and attention mechanism; averaging all tokens even improves global tagging and key estimation. The authors interpret this as emergent musical structure: per-column averaging of one chosen layer-9 attention map produces an onset detector with F-score 0.877, while spectral flux gives 0.720 and a randomly initialized model gives 0.501. They also show that intermediate-layer self-similarity matrices display harmonic block structure in shallow layers and rhythmic subdiagonals in deeper layers, indicating different layers capture different musical dimensions.

Load-bearing premise

The result rests on one hand-picked attention head standing in for the model's learned attention; if that head is atypical, the onset-detection evidence is a selected example rather than a systemic property.

Editorial extensions

If this is right

  • Simple contrastive pretraining can produce frame-level musical representations in sequence tokens without masked modeling or a projection head.
  • Averaging class and sequence tokens beats using the class token alone on global tasks, so the sequence tokens carry information not fully summarized by the class token.
  • Stacking intermediate-layer tokens improves harmonic tasks such as chord estimation, so shallow layers store complementary harmonic information.
  • A trained attention map serves as an onset detector at F-score 0.877, well above the spectral-flux baseline of 0.720 and the random-initialization value of 0.501.
  • A 5.3M-parameter contrastive model comes close to a much larger masked-modeling baseline on beat tracking (0.723 vs 0.794) and chord estimation (0.319 vs 0.322).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the class token's construction, so a direct test of its weight-sharing explanation would be to replace the average-based class token with a purely learnable vector and see whether the sequence-token gains disappear.
  • The head-to-head comparison with a much larger masked-modeling model is suggestive but not a benchmark; extending the same frozen-token probes to additional local tasks and datasets would show whether the gap is consistently small.
  • The layer-wise split between harmony and rhythm suggests a testable hierarchy in which shallow layers store harmonic frames and deeper layers store rhythmic patterns; one could probe intermediate tokens with specialized downstream heads to confirm it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies a lightweight Vision Transformer with one-dimensional time-frequency patches (ViT-1D) trained with a contrastive NT-Xent loss applied only to the class token. The authors report that, despite the class-token-level training objective, the sequence tokens acquire locally informative representations: probing the frozen sequence tokens yields a beat-tracking F-score of 0.723 and chord-estimation accuracy of 0.319, comparable to the much larger masked-modeling system M2D on the same evaluations. They also report that attention maps in deeper layers align with musical onsets (F-score 0.877 on MAPS-MUS, versus 0.501 for a randomly initialized model and 0.720 for spectral flux), and that self-similarity matrices of tokens at different layers capture harmonic versus rhythmic structure. The central claim is that contrastive SSL paired with a transformer exhibits emergent local musical properties that are commonly assumed to require masked modeling or other frame-level objectives.

Significance. If the empirical results hold, the paper challenges a widely held assumption in MIR that contrastive self-supervised representations are inadequate for local tasks such as beat tracking and chord estimation. The study is valuable because it provides a lightweight and conceptually simple alternative to large masked-modeling systems, and it contributes to the interpretability of audio transformers by connecting attention maps and self-similarity matrices to musical structure. The paper is strengthened by multiple converging lines of evidence: random-initialization controls are reported for downstream probing (Table 3), for onset detection (Table 2), and for SSM analysis (Section 6.1); the authors also release code and checkpoints. The main limitations concern the statistical robustness of the attention-map onset result and the lack of an ablation for the proposed weight-sharing mechanism.

major comments (3)
  1. [Section 5.2, Table 2] The onset-detection result, which is cited in the abstract and conclusion as evidence that onset events emerge from attention maps, rests on a single hand-picked attention head. The paper states 'We choose the attention matrix of an attention head from the 9th layer' and defends this with the assertion that 'similar properties are shown across many heads from 9th and 12th layers, also across multiple models initialized differently,' but no quantitative distribution is reported. If the selected head is atypical, the F-score of 0.877 versus 0.720 for spectral flux is a curated example rather than a systematic property. This is load-bearing for the representation-level emergence claim. The authors should report the distribution of onset F-scores across all heads and layers (mean, standard deviation, minimum, maximum) or specify an a priori selection rule. In addition, the 30 MAPS-MUS recordings are evaluated without confidence intervals or significance tests, so even the single-head number is statistically fragile. The downstream probing results in Tables 1 and 3 provide separate support for local-task competence, but they do not fix the specific claim about attention maps.
  2. [Sections 2 and 7] The proposed mechanism for the emergence of local properties is weight sharing between the class token and sequence tokens: the abstract says 'potentially thanks to weight sharing,' and the conclusion states that 'weight sharing and attention mechanisms allow temporal musical representations to emerge.' This causal claim is never ablated. A randomized or separately-parameterized class-token MLP, or a variant that removes the class token from the sequence entirely, would test whether weight sharing is actually necessary for the observed sequence-token competence. As written, the claim is speculative; the authors should either provide such an ablation or consistently hedge the causal language in the abstract and conclusion.
  3. [Section 5.2, peak-picking details] The onset-detection evaluation uses SciPy's peak-picking function but does not report the parameters used (height, prominence, distance), nor whether these parameters were tuned on the evaluation set. The pseudo-activation a(i) is a column average of a selected attention map, and the number and spacing of detected peaks can depend strongly on peak-picking hyperparameters. The authors should specify these parameters and report sensitivity to them, ideally by fixing them on a validation set before evaluating on MAPS-MUS. This is part of making the F-score comparison with spectral flux a fair and reproducible benchmark.
minor comments (7)
  1. [Equation (1)] The denominator of the NT-Xent loss does not include the temperature scaling in the exponent: the sum over negative samples should have exp(sim(...)/tau), matching the numerator. As written, the loss is not exactly the NT-Xent loss described in the text.
  2. [Section 1] There is a duplicated word in the pretext-task paragraph: 'We train ViT-1D with with a normalized temperature-scaled cross-entropy loss.'
  3. [Section 7 and Figure 4] There are typographical errors: 'self-similarity matrics' should be 'self-similarity matrices,' and the Figure 4 caption uses 'initializated' instead of 'initialized.'
  4. [Table 1] The table header has inconsistent spacing and capitalization ('MAP ROC W . ACC F- SCORE ACC', 'VIT-1D' vs 'ViT-1D'). Please reformat for readability.
  5. [Section 4] The M2D comparison is described as a reference, but Tables 1 and 2 might be misread as a controlled comparison. M2D is pretrained on different data and has substantially more parameters; the text should state more explicitly that the comparison is illustrative rather than a matched experimental condition.
  6. [Section 3.4] The chord-estimation dataset contains only 124 songs and the split is 8:1:1, but no random seed or repeated evaluation is reported. Adding the seed or reporting variance across splits would improve reproducibility.
  7. [Figures 2 and 4] The attention maps and SSMs are shown in grayscale with no colorbar. Adding colorbars and explicitly stating the value range in the caption would make the qualitative claims easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical observation that reports out-of-sample measurements, and none of its central claims reduces by construction to a fitted parameter or to a self-citation.

full rationale

The paper does not derive a result from an input that is defined in terms of the result. Its central claim is that sequence tokens of a ViT-1D trained with NT-Xent only on the class token perform well on local MIR tasks. This is supported by frozen-feature probing with freshly trained linear layers (Section 3 and Table 1), which is a genuine out-of-sample evaluation rather than a fitted quantity being renamed as a prediction. The attention-map onset detection (Section 5.2 and Table 2) is also an out-of-sample probe: the attention head is selected post hoc, but the F-score is measured against spectral flux and a random-initialization control, so the reported number is not equal to any training objective or fitted constant. The hand-picked attention head is a methodological weakness concerning selection bias and statistical robustness, but it is not circularity because the measurement does not reduce to the selection criterion by construction. The phrase 'potentially thanks to weight sharing' is an explicit hypothesis, not a citation-backed derivation. The self-citations (STONE, S-key, references [4] and [5]) appear only as related work on self-supervised pretext tasks and are not load-bearing for the paper's central argument. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The observation that random-initialized tokens reflect harmonic content from mel-spectrograms is acknowledged rather than presented as a trained emergent property. Overall, the paper is self-contained as an empirical study, and no step in its argument reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the 'emergent properties' are an empirical description of existing model components (sequence tokens, attention maps, self-similarity matrices). The ledger is dominated by domain assumptions about the pretext task, the input representation, and the probing protocol, plus three hand-chosen numerical constants that influence the reported scores. The absence of invented entities is a positive signal; the main epistemic cost is the unablated weight-sharing explanation and the proprietary pretraining corpus.

free parameters (3)
  • NT-Xent temperature tau = 0.1
    Hand-chosen temperature in Eq. 1. Standard in contrastive learning and not fitted to downstream performance, but it shapes the pretraining dynamics that produce the reported representations.
  • Beat tracking neighboring-frame smoothing value = 0.5
    Section 3.3: 'we increase the values of the two neighboring frames to 0.5 instead of 0'. Hand-chosen post-processing constant that directly affects the reported beat F-score of 0.723.
  • Onset detection peak-picking parameters = SciPy find_peaks defaults
    Section 5.2 uses SciPy peak picking on the averaged attention map without a sensitivity analysis; the choice of peak parameters affects the F=0.877 onset score.
assumptions (5)
  • domain assumption Two disjoint 4-second segments from the same song form positive pairs, and pulling their class tokens together is a useful pretext for musical representation learning.
    Section 2. This is the standard contrastive-SSL premise inherited from CLMR; if the pretext were musically vacuous, the downstream probing results would lack the claimed musical meaning.
  • domain assumption The mel-spectrogram already encodes harmonic structure, and skip connections propagate it into token representations.
    Section 6.1 explicitly acknowledges that a randomly initialized model's SSM resembles the mel-spectrogram and contains harmonic blocks; therefore only the rhythmic and onset structure is attributable to training, while harmonic 'emergence' in shallow layers partially inherits from the input design.
  • domain assumption Frame-wise linear probing on frozen features is an adequate measure of representation quality for local tasks.
    Sections 3.1 to 3.4 use single linear layers on frozen features; the conclusion that sequence tokens 'handle' local tasks depends on linear separability being a meaningful indicator of the representation's content.
  • domain assumption The chord vocabulary of 25 classes and the exclusion of unmappable chords do not bias the comparison.
    Section 3.4 excludes suspended and other chords, keeps one performance per song, and applies an 8:1:1 split over 124 songs; this preprocessing defines which sequences the 0.319 chord accuracy is measured on and may differ from the protocol used for the compared M2D reference.
  • standard math Standard transformer machinery (self-attention, NT-Xent, sinusoidal positional encoding) behaves as specified.
    Eqs. 1 and 2 invoke standard definitions from Vaswani et al. (ref [42]) and the ViT literature (ref [27]); no novel mathematics is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emergent musical properties of a transformer under contrastive self-supervised learning." pith.science (2026). https://pith.science/paper/ZQ7YVWPK

@misc{pith2026250623873,
  author       = {Pith},
  title        = {Pith review of: Emergent musical properties of a transformer under contrastive self-supervised learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQ7YVWPK}},
  note         = {Machine review of arXiv:2506.23873}
}
read the original abstract

In music information retrieval (MIR), contrastive self-supervised learning for general-purpose representation models is effective for global tasks such as automatic tagging. However, for local tasks such as chord estimation, it is widely assumed that contrastively trained general-purpose self-supervised models are inadequate and that more sophisticated SSL is necessary; e.g., masked modeling. Our paper challenges this assumption by revealing the potential of contrastive SSL paired with a transformer in local MIR tasks. We consider a lightweight vision transformer with one-dimensional patches in the time--frequency domain (ViT-1D) and train it with simple contrastive SSL through normalized temperature-scaled cross-entropy loss (NT-Xent). Although NT-Xent operates only over the class token, we observe that, potentially thanks to weight sharing, informative musical properties emerge in ViT-1D's sequence tokens. On global tasks, the temporal average of class and sequence tokens offers a performance increase compared to the class token alone, showing useful properties in the sequence tokens. On local tasks, sequence tokens perform unexpectedly well, despite not being specifically trained for. Furthermore, high-level musical features such as onsets emerge from layer-wise attention maps and self-similarity matrices show different layers capture different musical dimensions. Our paper does not focus on improving performance but advances the musical interpretation of transformers and sheds light on some overlooked abilities of contrastive SSL paired with transformers for sequence modeling in MIR.

Figures

Figures reproduced from arXiv: 2506.23873 by the authors.

Figure 2
Figure 2. Although we also study the 6th layer, it is omitted [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 53 canonical work pages

  1. [1]

    Emergent musical properties of a transformer under contrastive self-supervised learning

    INTRODUCTION We may categorize tasks in music information retrieval (MIR) as either local or global. Global tasks, such as music tagging and key estimation, are time-shift invariant and require a single prediction per piece of music. Local tasks, such as beat tracking and chord estimation, are time- © Y . Kong et al.. Licensed under a Creative Commons At-...

    work page 2025

  2. [2]

    CONTRASTIVE PRETEXT TASK Patching details: We compute the mel-frequency spectro- gram for a segment of duration equal to d = 4 seconds, obtaining matrices x, with 128 frequency bins and a frame rate of ξ = 31.5 Hz. Unlike standard ViT, which uses 2D patches, we extract 1D patches by taking all 128 mel bins from a single frame and apply one convolutional l...

    work page

  3. [3]

    no chord

    DOWNSTREAM TASKS We focus on two types of downstream tasks, commonly used in general-purpose SSL for MIR. We select music tag- ging and key estimation as representative global tasks and we choose beat tracking and chord estimation as examples of local tasks. A good performance on these four tasks re- quires the model to encode both harmonic and rhythmic r...

    work page

  4. [4]

    We also assess their contribution to global tasks

    RESULTS ON DOWNSTREAM TASKS Using the frozen output of the pretrained ViT-1D as input to a trainable linear layer for each task, we study whether sequence tokens capture local properties, despite the class token’s time-invariance. We also assess their contribution to global tasks. We compare this to two reference models, pretrained with contrastive learni...

    work page

  5. [5]

    ViT-1D has 12 layers in total

    PROPERTIES IN A TTENTION MAPS We study the emergent properties of tokens in the trans- former across different layers. ViT-1D has 12 layers in total. We select the 3rd, 6th, 9th, and 12th layers as repre- sentative points, as they are evenly spaced from shallower to deeper layers. A more comprehensive analysis of all 12 layers, as well as the potential pe...

    work page

  6. [6]

    ,zT k ] at layers k = 3, 6, 9, 12 (same as Section 5, denoted z3 to z12), along with tokens from a randomly initialized ViT-1D model, de- noted zr

    PROPERTIES IN SELF-SIMILARITY MA TRICS OF TOKENS 6.1 Qualitative analysis We extract intermediate tokens [z1 k, . . . ,zT k ] at layers k = 3, 6, 9, 12 (same as Section 5, denoted z3 to z12), along with tokens from a randomly initialized ViT-1D model, de- noted zr. For each zk, we compute a self-similarity matrix (SSM) Sk[i, j] = sim(zk[i], zk[j]) using c...

    work page

  7. [7]

    Applying NT-Xent loss only to the class token in a lightweight ViT-1D surprisingly enables sequence tokens to handle local tasks while con- tributing to global ones

    CONCLUSION In this paper, we show the ability of a general-purpose con- trastive pretext task paired with a transformer to learn lo- cal musical representations. Applying NT-Xent loss only to the class token in a lightweight ViT-1D surprisingly enables sequence tokens to handle local tasks while con- tributing to global ones. Despite the class token’s tim...

    work page

  8. [8]

    Contrastive learn- ing of musical representations,

    J. Spijkervet and J. A. Burgoyne, “Contrastive learn- ing of musical representations,” in Proc. of the Inter- national Society for Music Information Retrieval Con- ference (ISMIR), 2021

    work page 2021

Show all 55 references
  1. [9]

    In contrastive learning, the model is enforced by a loss to project positive pair samples close together in the embed- ding space and pushing negative samples far apart

    marked a first step forward, following the adoption of contrastive pretext task in computer vision [10, 11]. In contrastive learning, the model is enforced by a loss to project positive pair samples close together in the embed- ding space and pushing negative samples far apart...

    2025 arXiv

  2. [10]

    Convolutional operators in the time- frequency domain,

    V . Lostanlen, “Convolutional operators in the time- frequency domain,” Ph.D. dissertation, École normale supérieure, 2017

    2017

  3. [11]

    Pesto: Pitch estimation with self-supervised transposition-equivariant objective,

    A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “Pesto: Pitch estimation with self-supervised transposition-equivariant objective,” in Proc. of theInternational Society for Music Information Retrieval Conference (ISMIR), 2023

    2023

  4. [12]

    Equivariant self-supervision for musical tempo estimation,

    E. Quinton, “Equivariant self-supervision for musical tempo estimation,” inProc. of the International Society for Music Information Retrieval Conference (ISMIR) , 2022

    2022

  5. [13]

    STONE: Self- supervised tonality estimator,

    Y . Kong, V . Lostanlen, G. Meseguer-Brocal, S. Wong, M. Lagrange, and R. Hennequin, “STONE: Self- supervised tonality estimator,” Proc. of the Interna- tional Society for Music Information Retrieval Confer- ence (ISMIR), 2024

    2024

  6. [14]

    S-key: Self-supervised learning of major and minor keys from audio,

    Y . Kong, G. Meseguer-Brocal, V . Lostanlen, M. La- grange, and R. Hennequin, “S-key: Self-supervised learning of major and minor keys from audio,” in ICASSP 2025 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2025, pp. 1–5

    2025

  7. [15]

    Data cleansing with contrastive learning for vocal note event annotations,

    G. Meseguer-Brocal, R. Bittner, S. Durand, and B. Brost, “Data cleansing with contrastive learning for vocal note event annotations,” in Proceedings of the 21st International Society for Music Information Re- trieval Conference, 2020

    2020

  8. [16]

    Foundation models for music: A survey,

    Y . Ma, A. Øland, A. Ragni, B. M. Del Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Sha- tri et al. , “Foundation models for music: A survey,” arXiv preprint arXiv:2408.14340, 2024

    2024 arXiv

  9. [17]

    Supervised and un- supervised learning of audio representations for music understanding,

    M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. F. Ehmann, “Supervised and un- supervised learning of audio representations for music understanding,” 2022

    2022

  10. [18]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, “A simple framework for contrastive learning of visual representations,” CoRR, 2020

    2020

  11. [19]

    Exploring simple siamese rep- resentation learning,

    X. Chen and K. He, “Exploring simple siamese rep- resentation learning,” in Proc. of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  12. [20]

    S3t: Self-supervised pre-training with swin transformer for music classification,

    H. Zhao, C. Zhang, B. Zhu, Z. Ma, and K. Zhang, “S3t: Self-supervised pre-training with swin transformer for music classification,” in ICASSP 2022-2022 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP). IEEE, 2022, pp. 606–610

    2022

  13. [21]

    Multi- source contrastive learning from musical audio,

    C. Garoufis, A. Zlatintsi, and P. Maragos, “Multi- source contrastive learning from musical audio,” in Proc. of the Sound and Music Computing Conference (SMC), May 2023

    2023

  14. [22]

    On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations,

    M. C. McCallum, M. E. Davies, F. Henkel, J. Kim, and S. E. Sandberg, “On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal ...

    2024

  15. [23]

    Towards proper contrastive self-supervised learning strategies for mu- sic audio representation,

    J. Choi, S. Jang, H. Cho et al. , “Towards proper contrastive self-supervised learning strategies for mu- sic audio representation,” in 2022 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 2022, pp. 1–6

    2022

  16. [24]

    Jukebox: A generative model for music,

    P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” arXiv preprint arXiv:2005.00341, 2020

    2005 arXiv

  17. [25]

    Music2latent: Consistency autoencoders for latent audio compres- sion,

    M. Pasini, S. Lattner, and G. Fazekas, “Music2latent: Consistency autoencoders for latent audio compres- sion,” Proc. of the International Society for Music In- formation Retrieval Conference (ISMIR), 2024

    2024

  18. [26]

    Mert: Acoustic music understand- ing model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dan- nenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Y . Guo, and J. Fu, “Mert: Acoustic music understand- ing model with large-scale self-supervised training,” in Proc. of the...

    2023

  19. [27]

    Hu- bert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-S. Chuang et al. , “Hu- bert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans- actions on Audio, Speech, and Language Processing , vol. 29, pp. 3451–3460, 2021

    2021

  20. [28]

    Masked Modeling Duo: Towards a Universal Audio Pre-training Framework,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked Modeling Duo: Towards a Universal Audio Pre-training Framework,”IEEE/ACM Trans. Audio, Speech, Language Process. , vol. 32, pp. 2391–2406, 2024

    2024

  21. [29]

    A foundation model for music informatics,

    M. Won, Y .-N. Hung, and D. Le, “A foundation model for music informatics,” in Proc. of the IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

    2024

  22. [30]

    Towards learning universal audio repre- sentations,

    L. Wang, P. Luc, Y . Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.-B. Alayrac, S. Dieleman, J. Car- reira et al. , “Towards learning universal audio repre- sentations,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP...

    2022

  23. [31]

    Efficient training of audio transformers with patchout,

    K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Wid- mer, “Efficient training of audio transformers with patchout,” in Proc. Interspeech 2022, 2022, pp. 2753– 2757

    2022

  24. [32]

    AST: Audio Spectrogram Transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in Proc. of Interspeech , 2021, pp. 571–575

    2021

  25. [33]

    Contrastive audio-language learning for music,

    I. Manco, E. Benetos, E. Quinton, and G. Fazekas, “Contrastive audio-language learning for music,” in Proc. of the International Society for Music Informa- tion Retrieval Conference (ISMIR), 2022

    2022

  26. [34]

    MuLan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. Ellis, “MuLan: A joint embedding of music audio and natural language,” Proc. of the International Society for Music Information Retrieval Conference (ISMIR) , 2022

    2022

  27. [35]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in Interna- tional Conference on Le...

    2021

  28. [36]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proc. of the IEEE/CVF international conference on computer vi- sion, 2021, pp. 9650–9660

    2021

  29. [37]

    Dinov2: Learning robust visual features without su- pervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, R. Howes, P.-Y . Huang, H. Xu, V . Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatu...

    2023

  30. [38]

    An experimental comparison of multi-view self-supervised methods for music tagging,

    G. Meseguer-Brocal, D. Desblancs, and R. Hen- nequin, “An experimental comparison of multi-view self-supervised methods for music tagging,” in Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

    2024

  31. [39]

    Evaluation of algorithms using games: The case of music tagging,

    E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging,” inProc. of the International So- ciety for Music Information Retrieval Conference (IS- MIR), 2009, pp. 213–218

    2009

  32. [40]

    Sample-level deep convolutional neural networks for music auto- tagging using raw waveforms,

    J. Lee, J. Park, K. L. Kim, and J. Nam, “Sample-level deep convolutional neural networks for music auto- tagging using raw waveforms,” 2017

    2017

  33. [41]

    Fmak: A dataset of key and mode annotations for the free music archive– extended abstract,

    S. Wong and G. Hernandez, “Fmak: A dataset of key and mode annotations for the free music archive– extended abstract,” inProc. of the International Society for Music Information Retrieval Late-Breaking/Demo Session (ISMIR-LBD), 2023

    2023

  34. [42]

    Fma: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,”Proc. of theInternational Society for Music Information Re- trieval Conference (ISMIR), 2017

    2017

  35. [43]

    Two datasets for tempo estimation and key detection in electronic dance music annotated from user corrections,

    P. Knees, A. Faraldo, P. Herrera, R. V ogl, S. Böck, F. Hörschläger, and M. Le Goff, “Two datasets for tempo estimation and key detection in electronic dance music annotated from user corrections,” in Proc. of the International Society for Music Information Retrieval Conferenc...

    2015

  36. [44]

    Mir_eval: A transparent implementation of common mir metrics

    C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “Mir_eval: A transparent implementation of common mir metrics.” in Proc. of the International Society for Music Information Retrieval Conference (ISMIR) , vol. 10, 2014, p. 2014

    2014

  37. [45]

    A review of rhythm de- scription systems,

    F. Gouyon and S. Dixon, “A review of rhythm de- scription systems,” inProc. of the International Society for Music Information Retrieval Conference (ISMIR) , 2004

    2004

  38. [46]

    Gtzan- rhythm: Extending the gtzan test-set with beat, down- beat and swing annotations,

    U. Marchand, Q. Fresnel, and G. Peeters, “Gtzan- rhythm: Extending the gtzan test-set with beat, down- beat and swing annotations,” in Proc. of the Inter- national Conference on Music Information Retrieval Late-breaking/Demo (ISMIR-LBD), 2015

    2015

  39. [47]

    An efficient state- space model for joint tempo and meter tracking

    F. Krebs, S. Böck, and G. Widmer, “An efficient state- space model for joint tempo and meter tracking.” in Proc. of the International Society for Music Informa- tion Retrieval Conference (ISMIR), 2015, pp. 72–78

    2015

  40. [48]

    Schubert win- terreise dataset: A multimodal scenario for music anal- ysis,

    C. Weiß, F. Zalkow, V . Arifi-Müller, M. Müller, H. V . Koops, A. V olk, and H. G. Grohganz, “Schubert win- terreise dataset: A multimodal scenario for music anal- ysis,” Journal on Computing and Cultural Heritage (JOCCH), vol. 14, no. 2, pp. 1–18, 2021

    2021

  41. [49]

    Rwc music database: Popular, classical, and jazz music databases,

    M. Goto and H.Hashiguchi, “Rwc music database: Popular, classical, and jazz music databases,” Proc. of the International Conference on Music Information Re- trieval Conference (ISMIR), 2002

    2002

  42. [50]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural infor- mation processing systems, vol. 30, 2017

    2017

  43. [51]

    Sbert-wk: A sentence em- bedding method by dissecting bert-based word mod- els,

    B. Wang and C.-C. J. Kuo, “Sbert-wk: A sentence em- bedding method by dissecting bert-based word mod- els,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2146–2157, 2020

    2020

  44. [52]

    Multipitch esti- mation of piano sounds using a new probabilistic spec- tral smoothness principle,

    V . Emiya, R. Badeau, and B. David, “Multipitch esti- mation of piano sounds using a new probabilistic spec- tral smoothness principle,” IEEE Transactions on Au- dio, Speech, and Language Processing , vol. 18, no. 6, pp. 1643–1654, 2009

    2009

  45. [53]

    SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,

    P. Virtanen, R. Gommers, T. E. Oliphant, M. Haber- land, T. Reddy, D. Cournapeau, E. Burovski, P. Pe- terson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, ˙I. Polat, Y ....

    2020

  46. [54]

    How many layers and why? An analysis of the model depth in transformers,

    A. Simoulin and B. Crabbé, “How many layers and why? An analysis of the model depth in transformers,” in Proc. of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Pro- cessing: Student ...

    2021

  47. [55]

    What does bert look at? an analysis of bert’s attention,

    K. Clark, U. Khandelwal, O. Levy, and C. D. Man- ning, “What does bert look at? an analysis of bert’s attention,” in BlackBoxNLP@ACL, 2019

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.