Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

S-KEY: Self-supervised Learning of Major and Minor Keys from Audio

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Self-supervised key detection now matches supervised models

desk verdict First SSL method to match supervised key estimation, but the headline claim needs an overlap check and error bars before it fully stands. read the letter →

arxiv 2501.12907 v2 pith:56YSDYNW submitted 2025-01-22 cs.SD eess.AS

classification cs.SDeess.AS
keywords musickeyestimationself-supervisedlearningtonalitychromafeaturespseudo-labelsinformationretrievalcircleoffifthsmodeclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a self-supervised model, S-KEY, can estimate the full 24-key taxonomy (12 key signatures times major/minor mode) from raw audio with accuracy on par with the best supervised system, and that it is the first SSL method to do so. The approach extends the STONE architecture with an auxiliary pretext task that produces pseudo-labels for mode by comparing chroma energy at the estimated root with energy a minor third below it. Trained on 60,000 songs, S-KEY reaches a MIREX score of 72.1% on FMAKv2; scaling to one million unlabeled songs raises the score to 73.2%, matching the supervised state of the art at 73.1%. If right, this means expert musical annotations are not necessary for high-accuracy tonality estimation, and large unlabeled music corpora can substitute for them.

What carries the argument

The load-bearing mechanism is a structured output with two components: a pitch-equivariant vector \(\lambda_{\$\theta$,A,c}\) over the 12 pitch classes and a pitch-invariant vector \(\mu_{\$\theta$,A,c}\) over the two modes. A cross-power spectral density (CPSD) loss with \(\omega=7\) enforces equivariance to transposition along the circle of fifths, and a pseudo-labeling function \(\nu(\$\theta$|x,c)\) compares chroma energy at \(q_{\max}\) with energy a minor third below to generate a binary major/minor target. A batch-wise average loss \(L_{\text{avg}}\) steadies training by assuming roughly half major and half minor songs per batch, preventing cold-start collapse.

What would settle it

Train S-KEY on a corpus with known root annotations and compare qmax against the true root on songs where the CPSD loss is low; if qmax frequently selects the dominant or another scale degree instead of the tonic, the pseudo-labeling mechanism fails and the mode accuracy would drop, which would be visible as a large gap between S-KEY and a supervised model on mode-only accuracy.

Watch

Extended reading notes

Core claim

The central claim is that a self-supervised pretext task can replace human annotation for full 24-key tonality estimation. S-KEY combines the existing CPSD-based key-signature equivariance loss with a pseudo-labeling rule that decides major versus minor by comparing the pitch-class energy at the predicted root (qmax) against the energy three semitones below it, i.e., the root of the relative minor. Jointly optimizing key-signature equivariance and this pseudo-labeled mode objective yields a model that matches the supervised madmom network on FMAKv2, GTZAN, GiantSteps, and Schubert Winterreise, while using the same parameter budget as STONE and no labeled data.

Load-bearing premise

The method assumes that when the CPSD loss is low and a song is in a major key, the most-active pitch class qmax is the song's actual root note; if qmax points to the wrong scale degree, the pseudo-label for major versus minor is wrong.

Editorial extensions

If this is right

  • Self-supervised pretraining on unlabeled music can replace supervised training for tonality estimation, removing the need for expert key annotations.
  • Scaling the training corpus from 60k to 1M songs improves S-KEY on three of four evaluation datasets, showing that SSL benefits from larger unlabeled corpora.
  • The pseudo-labeling rule transfers to new datasets and genres without fine-tuning, and S-KEY outperforms supervised baselines on metal, jazz, and reggae.
  • Since S-KEY uses the same parameter budget as STONE, the additional mode discrimination comes from the pretext task design, not from added capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pseudo-labeling principle could be extended to other musical modes (e.g., blues or modal harmony) by choosing different interval comparisons below the root estimate, as the paper itself suggests.
  • Because S-KEY extracts a root estimate qmax on short segments, the architecture could be adapted to track key modulations over time within a single piece, a task the paper does not evaluate.
  • The reliance on a roughly balanced major/minor batch prior suggests the method might degrade on corpora with very skewed mode distributions, which could be tested by training on genre-specific collections.
  • The circle-of-fifths structure visible in the learned embeddings hints that the pretext task implicitly organizes harmonic relations; this could be probed as a general unsupervised representation for harmonic analysis beyond key classification.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces S-KEY, a self-supervised method for 24-class major/minor key estimation from audio. It extends STONE's CPSD-based pitch-equivariant pretraining with an auxiliary mode-classification task: a pseudo-label nu is computed by comparing the CQT energy at the predicted root qmax(theta|x) with the energy a minor third below, and the network is trained with binary cross-entropy against this pseudo-label together with a batch-level prior that roughly half the songs are major. S-KEY is trained on 60k or 1M songs from the Deezer catalog and evaluated on FMAKv2, GTZAN, GiantSteps, and SWD. The authors report MIREX scores that match or exceed the supervised state of the art (madmom) and conclude that S-KEY is the first SSL method to match supervised deep learning for 24-key tonality estimation.

Significance. If the claims hold, this is a meaningful advance: it would show that large-scale unlabeled audio can substitute for expert annotation in full-fledged key and mode estimation, while keeping the same parameter budget as STONE. The paper's strengths include a release of training and inference code, evaluation on four public benchmarks with a breakdown into MIREX, KSEA, and mode accuracy, and the nu-STONE ablation, which usefully isolates the contribution of the learned component over the rule-based heuristic. The main caveats are evidentiary: the headline comparisons rest on single-run point estimates and on a proprietary training corpus with no reported overlap control, and the pseudo-labeling mechanism is partly self-referential. These concerns do not diminish the interest of the idea but do affect whether the central claim is currently established.

major comments (4)
  1. [III-A, III-D, Table II] The training/evaluation separation is not audited. S-KEY is trained on 60k or 1M songs from the Deezer catalog (Section III-A) and evaluated on public recordings in FMAKv2, GTZAN, GiantSteps, and SWD (Section III-D), but the paper does not report any fingerprinting, deduplication, or overlap check between the training corpus and those evaluation sets. Because several evaluation sets consist of commercial or popular-music recordings that could plausibly appear in a large commercial catalog, even modest overlap could inflate the reported MIREX scores in Table II and would invalidate the claim that S-KEY matches supervised SOTA. This missing control is load-bearing for the headline result and must be supplied, or the claim should be restricted to a protocol that rules out overlap.
  2. [II-C, Equations (3)-(6)] The pseudo-labeling premise is self-referential and is stated as a postulate rather than tested. Equation (5) defines nu(theta|x,c) through qmax(theta|x) from Equation (3), and Equation (6) trains mu_theta toward nu; hence the network is trained toward targets it generated itself. The paper acknowledges the cold-start issue and introduces the batch prior in Equation (7), but it does not quantify how often qmax is correct, how sensitive the final result is to errors in qmax, or how much the pseudo-labels change during training. The claim that a known key signature plus the CQT energy rule reaches 79.4% mode accuracy is also stated without a derivation or confidence interval. Please report (a) agreement between qmax and available key annotations on the evaluation sets, (b) an ablation using an oracle qmax, and (c) statistics on pseudo-label stability over training.
  3. [IV-A, IV-B, Tables I and II] All reported scores are single-run point estimates with no error bars, number of seeds, or significance tests. The central numerical claims rest on margins that are within plausible sampling noise: S-KEY (60k) trails madmom by 1.0 MIREX point on FMAKv2 (72.1 vs 73.1), S-KEY (1M) leads by 0.1 point (73.2 vs 73.1), and the GTZAN lead is 6.5 points but comes from a single evaluation run. Without variance estimates, the words "matches" and "outperforms" are not established for these small margins. Please report per-seed means and standard deviations, confidence intervals obtained by bootstrapping over songs, or per-song score distributions.
  4. [III-B and Abstract] The abstract's claim "requiring no human annotation" is not fully supported as written. Section III-B describes creating two synthetic samples, one in C major and one in A minor, to calibrate the two output channels; this requires human music-theoretic knowledge to specify the correct labels for those samples. Although this is different from annotating a large training corpus, it is a form of supervision, and the paper should state this explicitly and explain why the calibration step does not count as annotation for the purpose of the headline claim.
minor comments (5)
  1. [II-F, Equation (8)] The text says the hyperparameters are lambda_BCE and lambda_avg, but the BCE term in Equation (8) is multiplied by lambda_S-KEY; please harmonize the notation.
  2. [IV-D] The word "unlabled" in "analyzing an unlabled corpus of 1M songs" should be "unlabeled."
  3. [Section I] The sentence "24-STONE, as a first proposition toward self-supervised key signature and mode estimator, underperforms by 15%" is vague; specify whether this is 15 percentage points on MIREX score and relative to which baseline.
  4. [Abstract and Section I] The abstract says S-KEY matches the supervised SOTA on FMAKv2 and GTZAN, but on FMAKv2 the 60k model trails madmom by 1.0 point and the match comes from the 1M model; please state that the match is obtained after scaling to 1M songs.
  5. [Table II note] The note that the reported madmom GiantSteps score differs from the original paper (74.6%) is helpful, but it would be even more useful to state the exact madmom version and parameters used for all datasets, not only for GiantSteps.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the SSL pseudo-label loop is self-referential by design but the headline benchmark is externally validated and not forced by construction.

full rationale

The derivation is self-contained against external benchmarks. S-KEY's mode pseudo-label ν(θ|x,c) in Eq. (5) is indeed a function of the network's pitch-equivariant output qmax(θ|x), and Eq. (6) trains the mode marginal µθ toward ν. This is a self-referential online pseudo-labeling objective, but it is not a circular reduction of the paper's central claim: the pseudo-label is not fitted to the evaluation labels, and the reported MIREX, KSEA, and mode accuracies are measured on human-annotated FMAKv2, GTZAN, GiantSteps, and SWD. The conditional postulate that low LCPSD makes qmax point to the root pitch class is an explicit assumption in Section II-C, inherited from the prior STONE work; it is empirically testable and is in fact tested through KSEA and mode accuracy. The self-citations to [16] are not used as an unverifiable uniqueness theorem; STONE is a published, externally evaluated system, so citing it as a building block does not make the headline result circular. The acknowledged 'cold start' problem in Section II-E is mitigated by the batch prior and calibration, and does not force the benchmark numbers. The reader's concern about training/evaluation overlap is a data-contamination risk rather than a circularity of the derivation chain, so it is outside this analysis.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

S-KEY depends on hand-set loss weights, a stated postulate about qmax, a 50/50 major/minor prior on training batches, and a calibration with two synthetic labeled examples. These are external ingredients beyond the public datasets and standard music theory.

free parameters (2)
  • lambda_BCE = 1.5
    Weight of the S-KEY pseudo-label loss in Eq. (8); hand-set so the three loss terms have similar magnitude at initialization.
  • lambda_avg = 15
    Weight of the batch-average mode-balance loss in Eq. (8); hand-set for the same initialization-magnitude reason.
assumptions (4)
  • ad hoc to paper For a song in a major key, when the CPSD loss is low, the pitch class with maximum lambda output is the root note.
    Stated as a postulate in Section II-C; the entire pseudo-labeling mechanism depends on qmax being correct.
  • ad hoc to paper Roughly half of the songs in every training mini-batch are major and half are minor.
    Section II-E introduces this to solve the cold start problem; it is enforced by the Lavg penalty but not verified against the Deezer corpus.
  • ad hoc to paper Two synthetic labeled examples, one in C major and one in A minor, are sufficient to calibrate the two output channels.
    Section III-B uses this calibration, which injects minimal external supervision into a pipeline described as annotation-free.
  • domain assumption The CQT energy at the major root versus the relative minor root is informative for mode, with a claimed 79.4% accuracy.
    Section II-C uses this acoustical heuristic to form weak pseudo-labels; no derivation or evidence is provided in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of S-KEY: Self-supervised Learning of Major and Minor Keys from Audio." pith.science (2026). https://pith.science/paper/56YSDYNW

@misc{pith2026250112907,
  author       = {Pith},
  title        = {Pith review of: S-KEY: Self-supervised Learning of Major and Minor Keys from Audio},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/56YSDYNW}},
  note         = {Machine review of arXiv:2501.12907}
}
read the original abstract

STONE, the current method in self-supervised learning for tonality estimation in music signals, cannot distinguish relative keys, such as C major versus A minor. In this article, we extend the neural network architecture and learning objective of STONE to perform self-supervised learning of major and minor keys (S-KEY). Our main contribution is an auxiliary pretext task to STONE, formulated using transposition-invariant chroma features as a source of pseudo-labels. S-KEY matches the supervised state of the art in tonality estimation on FMAKv2 and GTZAN datasets while requiring no human annotation and having the same parameter budget as STONE. We build upon this result and expand the training set of S-KEY to a million songs, thus showing the potential of large-scale self-supervised learning in music information retrieval.

Figures

Figures reproduced from arXiv: 2501.12907 by the authors.

Figure 1
Figure 1. Structured prediction: Summing yθ,A,c over rows produces a pitch￾equivariant component λθ,A,c, summing yθ,A,c per columns produces a pitch-invariant component µθ,A,c . Rows and columns are reversed in the figure compared to the main text due to space limitation for the figure. rows by Q semitones apart, and apply a softmax transformation over all QM = 24 entries. This yields a matrix yθ,A,c with Q = 12 rows and M = … view at source ↗
Figure 2
Figure 2. Comparison between the supervised state of the art (x-axis) and S-KEY [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. 2-D visualization of FMAKv2 songs in major and minor keys after [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Text-to-Music Models Really Follow Instructions? A Counterfactual Evaluation of Key and Beat Grouping

    cs.SD 2026-08 conditional novelty 6.0 of 10

    Using matched neutral and target-swap prompts, ACE-Step 1.5 and Stable Audio 3 show real key control and partial beat control, while LeVo2 does not, and much four-beat agreement is just the models' default output.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Richard Parncutt, Psychoacoustic foundations of major-minor tonality , MIT Press, 2024

  2. [2]

    The music information retrieval evaluation exchange: Some observations and insights,

    J Stephen Downie, Andreas F Ehmann, Mert Bay, and M Cameron Jones, “The music information retrieval evaluation exchange: Some observations and insights,” Advances in music information retrieval , pp. 93–115, 2010

  3. [3]

    Feature learning and deep architectures: New directions for music informatics,

    Eric J. Humphrey, Juan P. Bello, and Yann Le Cun, “Feature learning and deep architectures: New directions for music informatics,” Journal of Intelligent Information Systems , vol. 41, pp. 461–481, 2013

  4. [4]

    Audio self-supervised learning: A survey,

    Shuo Liu, Adria Mallol-Ragolta, Emilia Parada-Cabaleiro, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bj ¨orn W. Schuller, “Audio self-supervised learning: A survey,” Patterns, vol. 3, no. 12, 2022

  5. [5]

    AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley, “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2871–2883, 2024

  6. [6]

    BYOL for audio: Self-supervised learning for general- purpose audio representation,

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino, “BYOL for audio: Self-supervised learning for general- purpose audio representation,” in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN) , 2021

  7. [7]

    Contrastive learning of musical representations,

    Janne Spijkervet and John Ashley Burgoyne, “Contrastive learning of musical representations,” in Proceedings of the International Society for Music Information Retrieval (ISMIR) Conference , 2021

  8. [8]

    MERT: Acoustic music understanding model with large-scale self-supervised training,

    Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu, “MERT: Acoustic music understanding model with large-scale self-supervised training,” in Proceedings ...

Show all 31 references
  1. [9]

    SPICE: Self-supervised pitch estimation,

    Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Velimirovi ´c, “SPICE: Self-supervised pitch estimation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1118–1128, 2020

  2. [10]

    PESTO: Pitch estimation with self-supervised transposition-equivariant objective,

    Alain Riou, Stefan Lattner, Ga ¨etan Hadjeres, and Geoffroy Peeters, “PESTO: Pitch estimation with self-supervised transposition-equivariant objective,” in Proceedings from the International Society for Music Information Retrieval Conference (ISMIR) , 2023

  3. [11]

    Equivariant self-supervision for musical tempo estimation,

    Elio Quinton, “Equivariant self-supervision for musical tempo estimation,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR) , 2022

  4. [12]

    Adapting pitch- based self supervised learning models for tempo estimation,

    Antonin Gagner ´e, Slim Essid, and Geoffroy Peeters, “Adapting pitch- based self supervised learning models for tempo estimation,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 956–960

  5. [13]

    Zero- note samba: Self-supervised beat tracking,

    Dorian Desblancs, Vincent Lostanlen, and Romain Hennequin, “Zero- note samba: Self-supervised beat tracking,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023

  6. [14]

    Deep unsupervised drum transcription,

    Keunwoo Choi and Kyunghyun Cho, “Deep unsupervised drum transcription,” in Proceedings of the International Society for Music Information Retrieval (ISMIR) Conference , 2019

  7. [15]

    Learning multi-level representations for hierarchical music structure analysis,

    Morgan Buisson, Brian Mcfee, Slim Essid, and Helene-Camille Crayen- cour, “Learning multi-level representations for hierarchical music structure analysis,” in Proceedings of the International Society for Music Information Retrieval (ISMIR) , 2022

  8. [16]

    Stone: Self-supervised tonality estimator,

    Yuexuan Kong, Vincent Lostanlen, Gabriel Meseguer-Brocal, Stella Wong, Mathieu Lagrange, and Romain Hennequin, “Stone: Self-supervised tonality estimator,” International Society for Music Information Retrieval Conference (ISMIR), 2024

  9. [17]

    Genre-agnostic key clas- sification with convolutional neural networks,

    Filip Korzeniowski and Gerhard Widmer, “Genre-agnostic key clas- sification with convolutional neural networks,” in Proceedings of the International Society on Music Information Conference (ISMIR) , 2018

  10. [18]

    Krumhansl, Cognitive foundations of musical pitch , Oxford University Press, 2001

    Carol L. Krumhansl, Cognitive foundations of musical pitch , Oxford University Press, 2001

  11. [19]

    Madmom: A new Python audio and music signal processing library,

    Sebastian B ¨ock, Filip Korzeniowski, Jan Schl ¨uter, Florian Krebs, and Gerhard Widmer, “Madmom: A new Python audio and music signal processing library,” in Proceedings of the 24th ACM international conference on Multimedia , 2016, pp. 1174–1178

  12. [20]

    Fmak: A dataset of key and mode annotations for the free music archive–extended abstract,

    Stella Wong and Gandalf Hernandez, “Fmak: A dataset of key and mode annotations for the free music archive–extended abstract,” in Proc. of the International Society for Music Information Retrieval Late- Breaking/Demo Session (ISMIR-LBD) , 2023

  13. [21]

    FMA: A dataset for music analysis,

    Micha¨el Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson, “FMA: A dataset for music analysis,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2017

  14. [22]

    Genre-specific key profiles,

    Cian Brien and Alexander Lerch, “Genre-specific key profiles,” in Proceedings of the International Computer Music Association Conference (ICMC), 2015

  15. [23]

    Giantsteps key dataset,

    ´Angel Faraldo Peter Knees and Richard V ogl, “Giantsteps key dataset,” https://github.com/GiantSteps/giantsteps-key-dataset, 2015

  16. [24]

    Schubert winterreise dataset: A multimodal scenario for music analysis,

    Christof Weiß, Frank Zalkow, Vlora Arifi-M ¨uller, Meinard M ¨uller, Hendrik Vincent Koops, Anja V olk, and Harald G Grohganz, “Schubert winterreise dataset: A multimodal scenario for music analysis,” Journal on Computing and Cultural Heritage (JOCCH) , vol. 14, no. 2, pp. 1–18, 2021

  17. [25]

    mir eval: A Transparent Implementation of Common MIR Metrics.,

    Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel, “mir eval: A Transparent Implementation of Common MIR Metrics.,” in Proceedings of the International Society for Music Information Retrieval Conference (I...

  18. [26]

    An experimental comparison of multi-view self-supervised methods for music tagging,

    Gabriel Meseguer-Brocal, Dorian Desblancs, and Romain Hennequin, “An experimental comparison of multi-view self-supervised methods for music tagging,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 1141–1145

  19. [27]

    Andrew Jaffe, Something Borrowed Something Blue: Principles of Jazz Composition, Advance Music, 2011

  20. [28]

    Blues tonality,

    Ethan Hein, “Blues tonality,” https://www.ethanhein.com/wp/2014/ blues-tonality/, 2014

  21. [29]

    The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,

    Bob L Sturm, “The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,” arXiv preprint arXiv:1306.1461 , 2013

  22. [30]

    Tonal cognition,

    Carol L Krumhansl and Petri Toiviainen, “Tonal cognition,” Annals of the New York Academy of Sciences , vol. 930, no. 1, pp. 77–91, 2001

  23. [31]

    Learning the helix topology of musical pitch,

    Vincent Lostanlen, Sripathi Sridhar, Brian McFee, Andrew Farnsworth, and Juan Pablo Bello, “Learning the helix topology of musical pitch,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 11–15

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.