MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music

Scott H. Hawley

REVIEW 3 major objections 4 minor 34 references

Training a hierarchical encoder of piano-roll images with shift-equivariance objectives, SIGReg, and masked-embedding prediction produces music representations that reconstruct, classify, and generate without labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:48 UTC pith:AFF53AI5

load-bearing objection A useful SSL-for-music system whose headline equivariance claim is partly a convergence check, since the 'shift' augmentation changes content rather than cleanly translating it. the 3 major comments →

arxiv 2607.14537 v1 pith:AFF53AI5 submitted 2026-07-16 cs.SD cs.LGeess.AS

MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music

Scott H. Hawley This is my paper

classification cs.SD cs.LGeess.AS

keywords self-supervised learningsymbolic musicpiano rollequivarianceJEPASwin Transformerflow matchingmusic representation learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that a music encoder can learn rich, hierarchical representations of symbolic music without any labels by training it to be equivariant to pitch and time shifts. Music is encoded as 128x128 piano-roll images — time across the horizontal axis, the full 128 MIDI pitches up the vertical axis — and a Swin Transformer V2 encodes each 8-bar segment into six levels of abstraction. Trained only with self-supervised losses (equivariance, isotropic Gaussian regularization, and masked-embedding prediction), the frozen encoder embeddings support near-perfect reconstruction (F1 ≈ 0.995), outperform a scattering-transform baseline on emotion classification, and condition a flow-matching generator that tracks the pitch register and rhythmic density of the prompt. If the claim holds, it offers a path to musically meaningful representations for co-writing assistants without expensive labels or large-scale compute.

Core claim

The central claim is that equivariance-based self-supervised objectives, combined with a sufficiently capable fine-level encoder, produce representations of symbolic music that are simultaneously good for reconstruction, discriminative tasks, and generation. The strongest evidence is quantitative: a decoder trained on the frozen encoder embeddings reconstructs piano rolls at F1 ≈ 0.995; a linear probe on mid-level embeddings beats a scattering-transform baseline on a four-quadrant emotion classification benchmark; and a flow-matching model conditioned on the embeddings produces generations whose pitch register and rhythmic density match the conditioning excerpt, while mismatched conditioning

What carries the argument

The load-bearing mechanism is the equivariance objective with a delta-scaled target distance: pairs of embeddings from two shifted views of the same excerpt are pushed to a distance proportional to the shift magnitude, with a sqrt(d) scaling to make the target meaningful across hierarchy levels. Unlike a hinge loss, this smooth quadratic both attracts pairs that are too close and repels pairs that are too far, preventing collapse while keeping a geometrically consistent latent space. Collapse is further prevented by chunked SIGReg, which enforces an isotropic Gaussian prior on student embeddings; an EMA teacher supplies stable targets for the equivariance loss and for a masked embedding pred

Load-bearing premise

The load-bearing premise is that the shift operation implements a true geometric transformation; in fact, the second view is a different window of the same song sampled at an offset, so the equivariance loss compares different musical contexts rather than the same pattern under translation.

What would settle it

Train the same architecture with a true in-place translation: shift the contents of the first 128x128 crop and fill the vacated region with a constant (e.g., zero), then measure embedding distance versus shift magnitude. If monotonicity persists, equivariance reflects geometric transformation; if it flattens or changes shape, the reported curves are mostly content-dissimilarity artifacts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

The self-supervised objective alone forces the encoder to retain enough information for near-exact piano-roll reconstruction (F1 ≈ 0.995), so labels are not needed to get a near-lossless music code.
Learned embeddings transfer to a downstream emotion classification task without fine-tuning, beating a scattering-transform baseline, suggesting the representations capture musically relevant structure.
Conditioning a flow-matching generator on the embeddings gives control over pitch register and rhythmic density; shuffled conditioning produces unrelated but plausible music, confirming the embeddings carry meaningful information the generator exploits.
The soft factorization loss makes pitch and time directions nearly orthogonal in latent space, which could enable independent manipulation of pitch and rhythm in future generation or editing systems.
Because the encoder is trained on a compact dataset on consumer GPUs, the method is accessible without large-scale compute, though coarser hierarchy levels remain less interpretable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's 'shift' creates the second view by sampling a different 128x128 window at an offset from the same piano roll, meaning old content exits and new content enters; the equivariance curves may therefore partly reflect content dissimilarity rather than a geometric transformation of the same musical pattern. If so, the monotonic distance curves would still support a weaker claim: the encoder
The factorization loss's discovered orthogonal directions suggest a concrete testable extension: interpolating or stepping along the pitch direction in latent space should transpose a generated passage, and stepping along the time direction should shift its rhythm, enabling disentangled musical editing.
Since the PCA whitening discarded the mean pitch and the authors had to append it explicitly, other global descriptors (tempo, average note density) may similarly need explicit conditioning channels; a likely extension is to add those as auxiliary conditioning dimensions in the flow model.
Given piano rolls share a time-frequency grid with spectrograms, the same equivariance-plus-JEPA recipe is a natural candidate for self-supervised audio representation learning, provided the shift mechanism is adapted so that views are true transformations of the same content.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful SSL-for-music system whose headline equivariance claim is partly a convergence check, since the 'shift' augmentation changes content rather than cleanly translating it.

read the letter

Short version: this is a competent system paper for self-supervised symbolic music, with some genuinely new pieces, but the headline equivariance result is not as strong as it looks; the view-generation procedure confounds true geometric shift with content replacement. The downstream evidence (reconstruction, EMOPIA, conditioning-usage) is real and worth taking seriously.

The new stuff: a hierarchical Swin V2 encoder trained with LeJEPA/SIGReg, a delta-scaled equivariance loss that sets a target distance proportional to shift, a soft factorization loss for pitch/time directions, a frozen-encoder RAE decoder, and flow-matching conditioning on PCA-reduced embeddings. That combination is not in the cited literature, as far as I know. The decoder reaching F1 ≈ 0.995 shows the frozen embeddings retain enough detail; the EMOPIA linear probe beating Haar scattering is meaningful; the Wcond norm control in Fig. 8 is a nice touch because it shows the flow model actually uses the conditioning signal.

Where it gets soft: Fig. 6 and Table 2 mostly verify that the training loss was optimized. Of course embedding distance increases with shift magnitude when you train with L_eq, and cosine differences align when you train with L_fact. Those are convergence checks, not independent evidence. The bigger problem is the view generation in §2.3. The text says a shifted view is a second 128×128 window sampled at an offset (Δx, Δy) from the song's complete piano roll. That is not a translation of the same content; it is a different crop, with notes leaving one edge and new notes entering. With a fixed 128-pitch range, a vertical offset can't even be realized without either changing pitch content or having a larger canvas. So the "equivariance" loss may simply be teaching the encoder to be sensitive to crop position and overlap, and the monotonic curves may be measuring content dissimilarity. The factorization loss has the same confound, since the difference vectors come from different crops. This doesn't sink the reconstruction or generation results, but it hollows out the paper's central claim about equivariance-based SSL being the mechanism.

Other soft spots are minor: no error bars or seeds reported in Tables 3–5, the generation results are only qualitative (no FAD or other objective metric), and there's no commit hash on the GitHub link, which makes reproduction harder.

Who's this for? People working on SSL for symbolic music or on view-generation design for JEPA-style methods. It deserves a serious referee, but the revision needs to address the view-generation confound directly, add multiple seeds, and ideally quantify generation quality. If the authors can re-run with a true geometric shift (e.g., applying a pitch transposition and a time cut/shift, or roll the image), the equivariance narrative would be much cleaner. As written, it's a solid system paper with an overstated interpretation.

Referee Report

3 major / 4 minor

Summary. The paper introduces MIDI-RAE-JEPA, a self-supervised hierarchical representation learning method for symbolic music encoded as 128×128 binary piano rolls. The encoder is a 6-level Swin Transformer V2 trained with a combination of a pitch/time shift equivariance loss, SIGReg regularization, and a masked embedding predictor, with an optional soft factorization loss that encourages latent pitch and time directions to be orthogonal. A separately trained decoder reconstructs piano rolls from frozen embeddings with F1≈0.995, a linear probe on EMOPIA emotion classification outperforms a Haar scattering baseline, and a flow-matching generator conditioned on the embeddings tracks the pitch register and rhythmic density of the conditioning excerpt. The paper claims that equivariance-based SSL objectives, plus sufficient encoder capacity, yield semantically rich and generatively useful representations.

Significance. If the equivariance mechanism were sound, the paper would be a useful demonstration of a JEPA-style SSL objective for symbolic music, with a complete encode–decode–generate pipeline on consumer hardware and an explicit code release. The decomposition into encoder, decoder, and conditional generative model, and the use of a frozen-encoder linear-probe evaluation, follow current best practices in representation learning. The strongest independent evidence is the high decoder F1, the EMOPIA transfer result, and the conditioning-usage check in Figure 8; these would survive even if the equivariance interpretation were weakened. However, the central conceptual claim—that the model learns true pitch/time translation equivariance—is undermined by the view-generation procedure, as detailed below.

major comments (3)

[§2.3, View generation] The claimed 'shift' transformation is not a geometric translation of the input content. Both views are fixed 128×128 windows sampled from the song's complete piano roll at different offsets; a horizontal offset moves notes out of the left edge and introduces new notes at the right, and a vertical offset cannot be realized at all within the fixed 128-pitch range without changing pitch content. Thus x1 and x2 are partially overlapping but distinct musical contexts, not the same pattern under a translation. The equivariance loss Lequiv then trains the encoder to place different contexts at distances proportional to their spatial separation, so the monotonic curves in Figure 6 and the factorization geometry in Table 2 may simply reflect crop dissimilarity rather than learned geometric equivariance. This is a load-bearing issue because the title and abstract center on 'pitch- and time-shift e
[§3.1.1–3.1.2, Table 2 and Figure 6] The measurements presented as confirmations of equivariance and factorization are direct consequences of the training losses. Lequiv enforces a linear relationship between embedding distance and normalized shift magnitude; Lfact directly targets the cosine similarities reported in Table 2. Therefore, the monotonic distance curves and the ≈0/−/+1 cosine values are convergence checks, not independent evidence that the representation behaves equivariantly for unseen transformations. This is especially problematic in light of the view-generation confound above. The authors should treat these results as sanity checks and provide at least one independent test of equivariance—e.g., applying a true translation to a fixed musical pattern and measuring embedding distance—or remove the claim that these curves 'confirm measurable equivariance.'
[Tables 3–5] No error bars, multiple seeds, or significance tests are reported for any of the probe, reconstruction, or generation experiments. For example, Table 3 reports MRJ-12 L3 4-class accuracy of 0.488 versus Haar scattering 0.411; without variance estimates it is impossible to judge whether this gap is meaningful given typical run-to-run variation in SSL training. Table 5 explicitly states that the F1 values are 'close enough to be within the noise floor of training runs' yet gives no noise estimate. At minimum, the authors should report means and standard deviations over at least 3–5 seeds for the key comparisons, or state that the differences are not statistically significant.

minor comments (4)

[§1, Contributions] Typo: 'system forhierarchical SSL representations' should read 'system for hierarchical SSL representations.'
[§2.5, Generative Model] The PCA-with-whitening step retaining 95% variance and the explicit append of the mean pitch are described, but the exact PCA dimensionality after reduction for each hierarchy level is not reported. Adding this would help reproducibility.
[§3.1.3, Table 4] The DINOv2 baseline is compared only at its L0 and L1 (the two finest levels of its ViT hierarchy), while the proposed model is compared across all six levels. A fairer comparison would report DINOv2 features at all available levels with equivalent pooling and probe training.
[§2.4, Decoder] The decoder architecture is described as a 'mirrored stack of transformer stages with an FPN,' but key details (number of parameters, attention heads, exact skip-connection mechanism) are omitted. Since decoder F1 is a central result, a fuller architectural description is needed.

Circularity Check

2 steps flagged

Equivariance and factorization 'confirmations' are convergence checks: the losses define the target distances and cosine geometries that the results then report as validations.

specific steps

self definitional [§2.3 (Lequiv equation); §3.1.2 / Fig. 6; Abstract]
"Instead we enforce a target embedding distance proportional to the shift magnitude, so that larger shifts produce proportionally more distant embeddings: Lequiv(z1,z2,δ) = (∥z1−z2∥ − α√d∥δ̂∥)^2 ... embedding distances increase monotonically with pitch and time shift magnitude, confirming measurable equivariance."

The loss is a squared regression of embedding distance to a target α√d∥δ̂∥ that increases with shift magnitude. Once minimized, the measured distance-vs-shift curves in Fig. 6 are the training target, not an independent confirmation of equivariance. At levels where Lequiv is applied, monotonicity is imposed by construction; reporting it as 'confirming measurable equivariance' treats a convergence check as a prediction.
self definitional [§2.3 (Soft factorization); §3.1.1 / Table 2]
"Lfact = (cos(d1,d2)−t)^2 where the target t∈{+1,0,−1} encodes the geometric relationship dictated by the augmentation types: same-type same-sign shifts should be parallel (t=+1), same-type opposite-sign shifts anti-parallel (t=−1), and cross-type shifts (one pitch, one time) orthogonal (t=0). ... Table 2 shows that cross-type pairs ... are nearly orthogonal (|cos|<0.02), same-type same-sign pairs are strongly parallel (cos≈0.78), and same-type opposite-sign pairs show clear anti-parallel alignment (cos≈−0.5). The loss establishes these geometric relationships at L0–L2."

The table verifies the exact cosine targets imposed by Lfact, so the near-orthogonality/parallelism measures optimization success rather than evidence that pitch and time directions are naturally factored. The paper even states 'the loss establishes these geometric relationships,' acknowledging the measured geometry is a direct product of the training objective. Only unregularized L3 offers weak independent evidence via propagation through the hierarchy.

full rationale

Two advertised validations of the core equivariance/factorization narrative are convergence checks on the training objectives. L_eq regresses embedding distance onto α√d∥δ̂∥, so Fig. 6's monotonic curves restate the target for the trained levels; L_fact imposes the same cosine targets that Table 2 then reports. These do not reduce the entire paper: decoder F1 ≈ 0.995, EMOPIA linear-probe transfer, and the Wcond conditioning-weight norm are external to the equivariance loss and provide independent evidence that the learned embeddings retain usable musical information. Self-citation is not load-bearing here (LeJEPA/SIGReg are external works). A separate construct-validity concern, noted in §2.3, is that a 'shift' is implemented by sampling an offset window from the complete piano roll, which changes musical content rather than applying a true translation to the same content; this reinforces the circularity of the Fig. 6 confirmation but is not itself a self-citation or fitted-prediction step. Overall: partial circularity, score 6.

Axiom & Free-Parameter Ledger

7 free parameters · 7 axioms · 1 invented entities

The central claim rests on standard SSL machinery (SIGReg, concentration of measure), domain assumptions about piano-roll shifts, and several hand-chosen hyperparameters. The most fragile premise is the offset-crop view generation, which may confound content change with geometric transformation. The soft factorization geometry is an invented structure whose only direct evidence is the training objective itself.

free parameters (7)

alpha (equivariance distance scale) = not reported
Target embedding distance α√d‖δ̂‖ sets the scale of the latent metric; hand-chosen and never reported numerically.
Loss weights λ, λ_MEP, λ_fact = not reported
Balance equivariance, SIGReg, masked embedding prediction, and factorization objectives; values not given.
EMA momentum η = 0.96
Chosen for stable teacher targets; affects how slowly targets track the student.
Max shifts (Δtmax, Δpmax) = 12/12 or 48/12 pixels
MRJ-12 vs MRJ-48 changes training range and measurably alters reconstruction F1 and equivariance saturation.
Beta(2,2) shift distribution = Beta(2,2)
Overweights small shifts relative to uniform; this shape choice biases which equivariance relationships are learned best.
PCA 95% variance retention = 95%
Compresses conditioning embeddings; whitening discards the mean, so mean pitch is appended as an extra dimension.
Binarization threshold = 0.5
Converts decoder outputs to binary piano rolls; threshold choice affects F1.

axioms (7)

standard math SIGReg's Epps-Pulley statistic enforces an approximately isotropic Gaussian latent prior
Borrowed from LeJEPA; the paper relies on it to prevent collapse and to justify PCA/whitening behavior.
standard math Concentration of measure justifies the √d normalization in the equivariance loss
Used in L_eq to give α consistent meaning across hierarchy levels.
domain assumption Piano-roll rows/columns are meaningful pitch/time axes and translation along them preserves musical identity
The entire equivariance objective assumes shifting in time and pitch is a musically natural transformation with controllable structure.
ad hoc to paper An offset crop with the same window size is a valid implementation of a shift transformation
§2.3 view generation samples the second window at an offset; this may introduce new content rather than transforming the same content. This is the weakest load-bearing premise.
domain assumption POP909 plus random crop and transposition augmentation is enough to avoid overfitting
Only 909 songs are used; the authors argue augmentation expands effective diversity.
domain assumption Linear probes on mean-pooled embeddings measure transferable semantic content
Used for EMOPIA and structure probes; implicitly assumes mean pooling does not destroy relevant information.
domain assumption Flow matching with minibatch OT and multi-level dropout learns the correct conditional distribution
Adopted from prior work; conditioning evidence relies on Wcond and visual inspection.

invented entities (1)

Latent pitch/time axes (soft factorization geometry) no independent evidence
purpose: Factorize embedding space so pitch shifts and time shifts move along orthogonal directions; intended to support structured generation and motif analysis.
The directions are emergent rather than prescribed, but the only evidence (Table 2, Fig. 5) is cosine-similarity targets imposed by L_fact; no external probe shows these axes are musically interpretable.

pith-pipeline@v1.3.0-alltime-deepseek · 9717 in / 15769 out tokens · 175584 ms · 2026-08-02T01:48:10.792082+00:00 · methodology

0 comments

read the original abstract

Rich internal representations of musical structure are essential for music understanding tasks such as machine-assisted music co-writing, yet self-supervised approaches for symbolic music representation remain underexplored, particularly those that encode the hierarchical multiscale nature of musical structures. We present MIDI-RAE-JEPA, combining a pitch- and time-shift equivariance objective with LeJEPA and a Swin Transformer V2 encoder to learn such hierarchical representations of symbolic music encoded as piano roll images. The time-shift equivariance objective encourages the model to internalize temporal musical relationships. The encoder is trained purely on self-supervised objectives -- including a masked embedding predictor (MEP) -- with collapse prevented via SIGReg. A separate decoder trained on the frozen encoder embeddings achieves reconstruction F1 of 0.995, and a flow matching generative model conditioned on those embeddings produces generations that closely match the pitch register and rhythmic density of the conditioning excerpt, while mismatched conditioning yields unrelated but musically plausible output. Learned representations outperform a Haar scattering transform baseline on a downstream emotion classification task, and embedding distances increase monotonically with pitch and time shift magnitude, confirming measurable equivariance. These results suggest that equivariance-based SSL objectives, combined with sufficient fine-level encoder capacity, provide a viable path toward semantically rich, generatively useful representations of symbolic music.

Figures

Figures reproduced from arXiv: 2607.14537 by Scott H. Hawley.

discussion (0)

Reference graph

Works this paper leans on

34 extracted references · 9 linked inside Pith

[1]

zooming in

INTRODUCTION A long-standing goal in music technology is an intelligent AI assistant that can listen to a musical idea from a human and provide meaningful feedback to refine the composi- tion, arrangement, or production. Such a system requires rich internal representations of musical structure that cap- ture not only local note patterns but also the hiera...

Pith/arXiv arXiv 2026
[2]

Multi-Level Dropout

METHOD 2.1 Data Representation We use POP909 [18], 909 popular songs in MIDI format with separated melody and accompaniment. Each 8-bar segment is rendered as a 128×128 binary piano roll (union of melody and accompaniment), with 64 eighth- note time steps horizontally and all 128 MIDI pitches ver- tically. While larger MIDI corpora exist (e.g., the Lakh d...
[3]

RESULTS 3.1 Encoder Probes Figure 4 shows PCA scatter plots of mean-pooled embed- dings at each hierarchy level. Finer levels (L5, L4) exhibit tight, well-separated clusters reflecting local note content, while coarser levels (L1, L0) show broader, more isotropic distributions consistent with SIGReg’s Gaussian prior — the coarser levels encode larger-scal...
[4]

CONCLUSION We have shown that a self-supervised equivariance objec- tive, combined with a Swin V2 encoder and LeJEPA, pro- duces representations of symbolic music that are useful across multiple downstream tasks without any end-to-end training. A decoder trained on frozen encoder embeddings achieves F1 ≈ 0.995, confirming that the SSL objective alone forc...
[5]

Thanks to Vincent Lostanlen for the suggestion of the Haar wavelet baseline

ACKNOWLEDGEMENTS Thanks TwinOS and Razer Corporation for making avail- able the Blade 16” and Blade 18” GPU laptops used for some of the computations in this paper. Thanks to Vincent Lostanlen for the suggestion of the Haar wavelet baseline
[6]

AI USAGE STATEMENT A large language model (Claude, Anthropic) was used as a coding assistant for debugging, visualization, and test gen- eration; as an experiment management assistant for launch- ing jobs and tracking results; and as an interactive writing assistant in editing this paper, including portions of the introduction and related work. All scient...
[7]

Lerdahl and R

F. Lerdahl and R. Jackendoff,A Generative Theory of Tonal Music. MIT Press, 1983

1983
[8]

CMI-Bench: A comprehensive benchmark for evaluating music instruction following,

Y . Ma, S. Li, J. Yu, E. Benetos, and A. Maezawa, “CMI-Bench: A comprehensive benchmark for evaluating music instruction following,” inProceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR), 2025

2025
[9]

Polyffusion: A diffu- sion model for polyphonic score generation with internal and external controls,

L. Min, J. Jiang, G. Xia, and J. Zhao, “Polyffusion: A diffu- sion model for polyphonic score generation with internal and external controls,” inProceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR), 2023

2023
[10]

DINOv2: Learning robust visual features without supervision,

M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “DINOv2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

Pith/arXiv arXiv 2023
[11]

Self-supervised learning from images with a joint-embedding predictive architecture,

M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y . LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” inConference on Computer Vision and Pattern Recognition, 2023

2023
[12]

Diffusion trans- formers with representation autoencoders,

B. Zheng, N. Ma, S. Tong, and S. Xie, “Diffusion trans- formers with representation autoencoders,”arXiv preprint arXiv:2510.11690, 2025

Pith/arXiv arXiv 2025
[13]

LeJEPA: Provable and scalable self-supervised learning without the heuristics,

R. Balestriero and Y . LeCun, “LeJEPA: Provable and scalable self-supervised learning without the heuristics,” 2025. [Online]. Available: https://arxiv.org/abs/2511.08544

Pith/arXiv arXiv 2025
[14]

Emerging properties in self- supervised vision transformers,

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self- supervised vision transformers,” inInternational Conference on Computer Vision, 2021

2021
[15]

Masked modeling duo: Learning representations by encouraging both networks to model the input,

D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Learning representations by encouraging both networks to model the input,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

2023
[16]

V-jepa 2: Self-supervised video models enable understanding, prediction and planning,

M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, Mojtaba, Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V . Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma, S. Chandar, F. Meier, Y . LeCun, M. Rabbat, and N. Ballas, “V-jepa 2: Self-supe...

Pith/arXiv arXiv 2025
[17]

PESTO: Pitch estimation with self-supervised transposition-equivariant objective,

A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “PESTO: Pitch estimation with self-supervised transposition-equivariant objective,” inInternational Society for Music Information Retrieval Conference, 2023

2023
[18]

Equivariant self-supervision for musical tempo estimation,

E. Quinton, “Equivariant self-supervision for musical tempo estimation,” inProceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR), 2022, pp. 84–92

2022
[19]

STONE: Self-supervised tonality estimator,

Y . Kong, V . Lostanlen, G. Meseguer-Brocal, S. Wong, M. La- grange, and R. Hennequin, “STONE: Self-supervised tonality estimator,” inProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024, pp. 954–961

2024
[20]

Toward fully self-supervised multi-pitch estimation,

F. Cwitkowitz and Z. Duan, “Toward fully self-supervised multi-pitch estimation,”arXiv preprint arXiv:2402.15569, 2024

Pith/arXiv arXiv 2024
[21]

MusicBERT: Symbolic music understanding with large-scale pre-training,

M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y . Liu, “MusicBERT: Symbolic music understanding with large-scale pre-training,” inFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 791–800

2021
[22]

MidiBERT-Piano: Large-scale pre-training for symbolic mu- sic classification tasks,

Y .-H. Chou, I.-C. Chen, J. Ching, C.-J. Chang, and Y .-H. Yang, “MidiBERT-Piano: Large-scale pre-training for symbolic mu- sic classification tasks,”Journal of Creative Music Systems, vol. 8, no. 1, 2024

2024
[23]

MuseBERT: Pre-training music repre- sentation for music understanding and controllable generation,

Z. Wang and G. Xia, “MuseBERT: Pre-training music repre- sentation for music understanding and controllable generation,” inProceedings of the 22nd International Society for Music In- formation Retrieval Conference (ISMIR), 2021, pp. 722–729

2021
[24]

POP909: A pop-song dataset for music arrangement generation,

Z. Wang, K. Chen, J. Jiang, Y . Zhang, M. Xu, S. Dai, G. Xia, and S. Dubnov, “POP909: A pop-song dataset for music arrangement generation,”arXiv preprint arXiv:2008.07142, 2020

Pith/arXiv arXiv 2008
[25]

Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching,

C. Raffel, “Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching,” Ph.D. dissertation, Columbia University, 2016

2016
[26]

Swin transformer: Hierarchical vision transformer using shifted windows,

Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inInternational Conference on Com- puter Vision, 2021

2021
[27]

Swin transformer V2: Scaling up capacity and resolution,

Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Donget al., “Swin transformer V2: Scaling up capacity and resolution,” inConference on Computer Vision and Pattern Recognition, 2022

2022
[28]

MelodyGLM: Multi-task pre- training for symbolic melody generation,

X. Wu, Z. Huang, K. Zhang, J. Yu, X. Tan, T. Zhang, Y . Li, Z. Wang, and L. Sun, “MelodyGLM: Multi-task pre- training for symbolic melody generation,”arXiv preprint arXiv:2309.10738, 2023

Pith/arXiv arXiv 2023
[29]

Music SketchNet: Control- lable music generation via factorized representations of pitch and rhythm,

K. Chen, G. Xia, and S. Dubnov, “Music SketchNet: Control- lable music generation via factorized representations of pitch and rhythm,”arXiv preprint arXiv:2008.01291, 2020

Pith/arXiv arXiv 2008
[30]

Flow matching for generative modeling,

Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview. net/forum?id=PqvMRDCJT9t

2023
[31]

Improving and generalizing flow-based generative models with minibatch optimal transport,

A. Tong, K. FATRAS, N. Malkin, G. Huguet, Y . Zhang, J. Rector-Brooks, G. Wolf, and Y . Bengio, “Improving and generalizing flow-based generative models with minibatch optimal transport,”Transactions on Machine Learning Research, 2024, expert Certification. [Online]. Available: https://openreview.net/forum?id=CD9Snc73AW

2024
[32]

Classifier-free diffusion guidance,

J. Ho and T. Salimans, “Classifier-free diffusion guidance,” inNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. [Online]. Available: https://openreview.net/forum?id=qw8AKxfYbI

2021
[33]

EMOPIA: A multi-modal pop piano dataset for emo- tion recognition and emotion-based music generation,

H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y .-H. Yang, “EMOPIA: A multi-modal pop piano dataset for emo- tion recognition and emotion-based music generation,” inProc. Int. Society for Music Information Retrieval Conf., 2021

2021
[34]

Unsupervised deep haar scattering on graphs,

X. Chen, X. Cheng, and S. Mallat, “Unsupervised deep haar scattering on graphs,” inAdvances in Neural Infor- mation Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Avail- able: https://proceedings.neurips.cc/paper_files/paper/2014/ file/34fde01345258939e718af...

2014

[1] [1]

zooming in

INTRODUCTION A long-standing goal in music technology is an intelligent AI assistant that can listen to a musical idea from a human and provide meaningful feedback to refine the composi- tion, arrangement, or production. Such a system requires rich internal representations of musical structure that cap- ture not only local note patterns but also the hiera...

Pith/arXiv arXiv 2026

[2] [2]

Multi-Level Dropout

METHOD 2.1 Data Representation We use POP909 [18], 909 popular songs in MIDI format with separated melody and accompaniment. Each 8-bar segment is rendered as a 128×128 binary piano roll (union of melody and accompaniment), with 64 eighth- note time steps horizontally and all 128 MIDI pitches ver- tically. While larger MIDI corpora exist (e.g., the Lakh d...

[3] [3]

RESULTS 3.1 Encoder Probes Figure 4 shows PCA scatter plots of mean-pooled embed- dings at each hierarchy level. Finer levels (L5, L4) exhibit tight, well-separated clusters reflecting local note content, while coarser levels (L1, L0) show broader, more isotropic distributions consistent with SIGReg’s Gaussian prior — the coarser levels encode larger-scal...

[4] [4]

CONCLUSION We have shown that a self-supervised equivariance objec- tive, combined with a Swin V2 encoder and LeJEPA, pro- duces representations of symbolic music that are useful across multiple downstream tasks without any end-to-end training. A decoder trained on frozen encoder embeddings achieves F1 ≈ 0.995, confirming that the SSL objective alone forc...

[5] [5]

Thanks to Vincent Lostanlen for the suggestion of the Haar wavelet baseline

ACKNOWLEDGEMENTS Thanks TwinOS and Razer Corporation for making avail- able the Blade 16” and Blade 18” GPU laptops used for some of the computations in this paper. Thanks to Vincent Lostanlen for the suggestion of the Haar wavelet baseline

[6] [6]

AI USAGE STATEMENT A large language model (Claude, Anthropic) was used as a coding assistant for debugging, visualization, and test gen- eration; as an experiment management assistant for launch- ing jobs and tracking results; and as an interactive writing assistant in editing this paper, including portions of the introduction and related work. All scient...

[7] [7]

Lerdahl and R

F. Lerdahl and R. Jackendoff,A Generative Theory of Tonal Music. MIT Press, 1983

1983

[8] [8]

CMI-Bench: A comprehensive benchmark for evaluating music instruction following,

Y . Ma, S. Li, J. Yu, E. Benetos, and A. Maezawa, “CMI-Bench: A comprehensive benchmark for evaluating music instruction following,” inProceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR), 2025

2025

[9] [9]

Polyffusion: A diffu- sion model for polyphonic score generation with internal and external controls,

L. Min, J. Jiang, G. Xia, and J. Zhao, “Polyffusion: A diffu- sion model for polyphonic score generation with internal and external controls,” inProceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR), 2023

2023

[10] [10]

DINOv2: Learning robust visual features without supervision,

M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “DINOv2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

Pith/arXiv arXiv 2023

[11] [11]

Self-supervised learning from images with a joint-embedding predictive architecture,

M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y . LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” inConference on Computer Vision and Pattern Recognition, 2023

2023

[12] [12]

Diffusion trans- formers with representation autoencoders,

B. Zheng, N. Ma, S. Tong, and S. Xie, “Diffusion trans- formers with representation autoencoders,”arXiv preprint arXiv:2510.11690, 2025

Pith/arXiv arXiv 2025

[13] [13]

LeJEPA: Provable and scalable self-supervised learning without the heuristics,

R. Balestriero and Y . LeCun, “LeJEPA: Provable and scalable self-supervised learning without the heuristics,” 2025. [Online]. Available: https://arxiv.org/abs/2511.08544

Pith/arXiv arXiv 2025

[14] [14]

Emerging properties in self- supervised vision transformers,

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self- supervised vision transformers,” inInternational Conference on Computer Vision, 2021

2021

[15] [15]

Masked modeling duo: Learning representations by encouraging both networks to model the input,

D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Learning representations by encouraging both networks to model the input,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

2023

[16] [16]

V-jepa 2: Self-supervised video models enable understanding, prediction and planning,

M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, Mojtaba, Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V . Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma, S. Chandar, F. Meier, Y . LeCun, M. Rabbat, and N. Ballas, “V-jepa 2: Self-supe...

Pith/arXiv arXiv 2025

[17] [17]

PESTO: Pitch estimation with self-supervised transposition-equivariant objective,

A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “PESTO: Pitch estimation with self-supervised transposition-equivariant objective,” inInternational Society for Music Information Retrieval Conference, 2023

2023

[18] [18]

Equivariant self-supervision for musical tempo estimation,

E. Quinton, “Equivariant self-supervision for musical tempo estimation,” inProceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR), 2022, pp. 84–92

2022

[19] [19]

STONE: Self-supervised tonality estimator,

Y . Kong, V . Lostanlen, G. Meseguer-Brocal, S. Wong, M. La- grange, and R. Hennequin, “STONE: Self-supervised tonality estimator,” inProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024, pp. 954–961

2024

[20] [20]

Toward fully self-supervised multi-pitch estimation,

F. Cwitkowitz and Z. Duan, “Toward fully self-supervised multi-pitch estimation,”arXiv preprint arXiv:2402.15569, 2024

Pith/arXiv arXiv 2024

[21] [21]

MusicBERT: Symbolic music understanding with large-scale pre-training,

M. Zeng, X. Tan, R. Wang, Z. Ju, T. Qin, and T.-Y . Liu, “MusicBERT: Symbolic music understanding with large-scale pre-training,” inFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 791–800

2021

[22] [22]

MidiBERT-Piano: Large-scale pre-training for symbolic mu- sic classification tasks,

Y .-H. Chou, I.-C. Chen, J. Ching, C.-J. Chang, and Y .-H. Yang, “MidiBERT-Piano: Large-scale pre-training for symbolic mu- sic classification tasks,”Journal of Creative Music Systems, vol. 8, no. 1, 2024

2024

[23] [23]

MuseBERT: Pre-training music repre- sentation for music understanding and controllable generation,

Z. Wang and G. Xia, “MuseBERT: Pre-training music repre- sentation for music understanding and controllable generation,” inProceedings of the 22nd International Society for Music In- formation Retrieval Conference (ISMIR), 2021, pp. 722–729

2021

[24] [24]

POP909: A pop-song dataset for music arrangement generation,

Z. Wang, K. Chen, J. Jiang, Y . Zhang, M. Xu, S. Dai, G. Xia, and S. Dubnov, “POP909: A pop-song dataset for music arrangement generation,”arXiv preprint arXiv:2008.07142, 2020

Pith/arXiv arXiv 2008

[25] [25]

Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching,

C. Raffel, “Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching,” Ph.D. dissertation, Columbia University, 2016

2016

[26] [26]

Swin transformer: Hierarchical vision transformer using shifted windows,

Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inInternational Conference on Com- puter Vision, 2021

2021

[27] [27]

Swin transformer V2: Scaling up capacity and resolution,

Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Donget al., “Swin transformer V2: Scaling up capacity and resolution,” inConference on Computer Vision and Pattern Recognition, 2022

2022

[28] [28]

MelodyGLM: Multi-task pre- training for symbolic melody generation,

X. Wu, Z. Huang, K. Zhang, J. Yu, X. Tan, T. Zhang, Y . Li, Z. Wang, and L. Sun, “MelodyGLM: Multi-task pre- training for symbolic melody generation,”arXiv preprint arXiv:2309.10738, 2023

Pith/arXiv arXiv 2023

[29] [29]

Music SketchNet: Control- lable music generation via factorized representations of pitch and rhythm,

K. Chen, G. Xia, and S. Dubnov, “Music SketchNet: Control- lable music generation via factorized representations of pitch and rhythm,”arXiv preprint arXiv:2008.01291, 2020

Pith/arXiv arXiv 2008

[30] [30]

Flow matching for generative modeling,

Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview. net/forum?id=PqvMRDCJT9t

2023

[31] [31]

Improving and generalizing flow-based generative models with minibatch optimal transport,

A. Tong, K. FATRAS, N. Malkin, G. Huguet, Y . Zhang, J. Rector-Brooks, G. Wolf, and Y . Bengio, “Improving and generalizing flow-based generative models with minibatch optimal transport,”Transactions on Machine Learning Research, 2024, expert Certification. [Online]. Available: https://openreview.net/forum?id=CD9Snc73AW

2024

[32] [32]

Classifier-free diffusion guidance,

J. Ho and T. Salimans, “Classifier-free diffusion guidance,” inNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. [Online]. Available: https://openreview.net/forum?id=qw8AKxfYbI

2021

[33] [33]

EMOPIA: A multi-modal pop piano dataset for emo- tion recognition and emotion-based music generation,

H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y .-H. Yang, “EMOPIA: A multi-modal pop piano dataset for emo- tion recognition and emotion-based music generation,” inProc. Int. Society for Music Information Retrieval Conf., 2021

2021

[34] [34]

Unsupervised deep haar scattering on graphs,

X. Chen, X. Cheng, and S. Mallat, “Unsupervised deep haar scattering on graphs,” inAdvances in Neural Infor- mation Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Avail- able: https://proceedings.neurips.cc/paper_files/paper/2014/ file/34fde01345258939e718af...

2014