Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Predicting sound from action yields a music world model that beats passive listening on core MIR tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:05 UTC pith:SQAN6YQT

load-bearing objection A promising action-conditioned JEPA for piano with useful benchmarks, but the causal world-model and planning claims outrun the evidence. the 4 major comments →

arxiv 2607.22000 v1 pith:SQAN6YQT submitted 2026-07-24 cs.SD

Music-JEPA: Learning a World Model of Sound from Action

Ziyu Wang , Kun Fang , Yann LeCun This is my paper
classification cs.SD
keywords music world modeljoint embedding predictive architectureaction-conditioned dynamicslatent predictionpiano transcriptionself-supervised learningbeat trackingsustain pedal estimation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Music-JEPA asks whether a music model should be learned by predicting the future audio from the current audio plus the pianist's action, rather than by passively watching audio. The paper argues yes: with piano audio as state and the aligned pianoroll and sustain pedal as action, a JEPA-style latent predictor learns dynamics that connect actions to their acoustic consequences. The resulting representations outperform an audio-only JEPA on beat tracking, composer identification, and key recognition, and they enable a transcription method that plans by searching for actions that best explain a target sound. If the claim holds, it turns self-supervised music learning from passive observation into an interactive, causal problem.

Core claim

The central claim is that a high-dimensional instrument control can serve as the action variable in a latent world model, and that the paired transition st+1 = f(st, at+1) trained offline on aligned audio–MIDI segments captures genuine action–sound relationships. This is demonstrated three ways: (1) the model is more sensitive to perturbed state–action pairs than an audio-only baseline, with win rates near 1 for most temporal and random perturbations; (2) frozen state encoders transfer to beat tracking, composer identification, and key recognition, outperforming the audio-only baseline and staying competitive with a much larger pretrained model; (3) a separate inverse predictor and action de

What carries the argument

The engine is an action-conditioned latent transition: st+1 = f(st, at+1) with at+1 = g(at), where a state encoder maps a 2-second log-mel spectrogram to latent patches, an action encoder maps the aligned pianoroll and pedal to latent patches, and the state predictor f applies cross-attention to the action tokens while predicting a masked version of st+1. Training uses stop-gradient EMA teacher encoders and layer normalization to prevent collapse. A second, amortized module — an inverse predictor h(st, st+1, at) plus a frozen action decoder — inverts the dynamics for planning-based transcription.

Load-bearing premise

The model assumes that each 2-second audio segment is causally produced by the 2-second pianoroll and pedal segment aligned with it; in real piano sound, notes and sustain from earlier segments continue ringing into later ones, so this segment-local assumption may not match the true generative process.

What would settle it

Record piano audio where the sustain pedal is pressed during segment A and released at the exact boundary entering segment B, while the notes themselves stop at the boundary. A segment-local model cannot know the pedal state from the previous segment, so its predicted state for B will be wrong; if the model still predicts B correctly, it must have learned carry-over implicitly, and the segment-local assumption is not as limiting. Conversely, if it fails, that assumption is the weak point.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Action-conditioned latent dynamics capture temporal structure that passive masked prediction misses, as shown by much higher win rates when states or actions are perturbed.
  • The same learned state encoder supports three MIR tasks — beat tracking, composer identification, and key recognition — and does so at roughly 7% the parameter count of a larger pretrained model while staying competitive.
  • Planning-based transcription works: the model can find actions that explain a target audio state, and its continuous sustain-pedal estimates are more accurate than dedicated supervised pedal models.
  • The counterfactual synthesis results indicate the dynamics are functional: swapping in a different action changes the predicted spectrogram in a musically coherent way.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the transition is learned offline from aligned pairs, the same recipe may extend to non-piano instruments or to abstract 'actions' such as chord symbols or performance instructions, wherever paired state–action data exists.
  • The strong pedal result hints that actions with direct, reversible acoustic effects are especially learnable; this predicts that adding other continuous performance controls (tempo, loudness, articulation) to the action space should improve the model's predictive sensitivity.
  • A segment-local causality test would be to train on longer context or add an explicit carry-over state; if the current model's advantage on pedal estimation shrinks when notes are sustained across segment boundaries, the assumption of segment-local generation is likely the limiting factor.
  • The planning formulation suggests transcription could be inverted into interactive tools — e.g., editing a pianist's actions and hearing the consequence — without retraining the dynamics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Music-JEPA models piano audio as an action-conditioned dynamical system: 2-second audio segments are treated as states, paired MIDI pianoroll/pedal segments as actions, and two latent encoders plus predictors f and g are trained with an EMA objective (Eq. 3). The paper reports perturbation win rates (Fig. 2, Table 1), downstream MIR probing results (Table 2), and a transcription experiment framed as planning (Table 3). The central claim is that the learned model captures how musical actions produce sound and that the representation supports beat tracking, composer identification, key recognition, and transcription-style planning. The architecture is a sensible extension of JEPA to symbolic-audio paired data, but the current evidence for the world-model and planning claims is not conclusive: the main dynamics evaluation uses the same MSE as the training loss, the planning experiment bypasses the learned forward dynamics, and several experimental comparisons lack statistical error bars.

Significance. If the central claims held, action-conditioned JEPA would provide a compact, self-supervised route to a music world model that could be useful for prediction, representation learning, and planning. The paper introduces a novel training setup for paired audio-pianoroll data and provides a clean comparison against audio-only JEPA and MERT, with a relatively small model. It also makes an explicit attempt at planning-based transcription. These are worthwhile contributions. However, the current results do not yet establish causal generalization of the dynamics, nor that the learned dynamics enable planning; those are the core claims of the paper and need additional evidence.

major comments (4)
  1. [§4.3, Eq. (6) vs. Eq. (3)] The perturbation analysis uses the same state-prediction MSE that is minimized during training. High win rates for the correct triplet are therefore expected for any model that fits the training objective; they do not independently establish that the model captures a causal world model. The perturbations are also local and in-distribution (temporal shifts within the same performance, pitch/velocity/pedal scaling of existing actions). To support the claim, please evaluate with a metric that is not the training loss, e.g., quantitative spectrogram reconstruction against simulated ground truth, and include tests on genuinely novel action-state combinations. Also clarify whether the AO-JEPA baseline is evaluated with its own objective or with Eq. (6), since the two models are trained with different losses.
  2. [§4.5, Eq. (5)] The section is titled 'Transcription via Planning', but Eq. (5) is never solved. Instead, the authors train an inverse predictor h(s_t, s_{t+1}, a_t) on ground-truth actions and decode its outputs with a separate decoder; the learned forward dynamics f and action predictor g are not used for action search. Table 3 therefore evaluates a supervised inverse model/decoder, not the planning capability of the world model. The claim that 'learned dynamics enable planning' is unsupported. Please either optimize actions against Eq. (5) (or a tractable approximation), or clearly reframe the experiment as action inference without planning and adjust the abstract/conclusion accordingly.
  3. [Footnote 2 / Eq. (1)] The dynamical-system assumption is segment-local: 'a_{t+1} produces s_{t+1}'. For piano audio this is not exact: notes and sustain-pedal effects from earlier segments carry over into s_{t+1}, and pedal affects notes across segment boundaries. Because the paired training data always contain the true previous actions, f may fit correlations between a_t and the tail of s_t without learning the causal effect of a_{t+1}. The paper should address acoustic carry-over, e.g., by using overlapping segments, conditioning on previous actions, or validating on synthetic examples with controlled note offsets and pedal changes where ground-truth counterfactuals are known.
  4. [Table 2, §4.4] All downstream results are single-seed point estimates without error bars or significance tests. Several head-to-head comparisons are close—e.g., Ours vs. AO-JEPA beat F1@70 is 0.6208 vs. 0.6013, and key wF1 is 0.7617 vs. 0.7615—so the claim that Music-JEPA 'consistently outperforms' AO-JEPA is not statistically supported. In addition, the pseudo key labels are constructed by matching ground-truth pianoroll chroma to major scales; this should be disclosed as a limitation of the key-recognition evaluation. Please report mean and standard deviation over multiple seeds and perform significance tests for the headline comparisons.
minor comments (5)
  1. [Abstract, §4.2] The paper says Music-JEPA uses 'only 7% of MERT's parameters', but the full Music-JEPA model has 19M parameters, roughly 20% of MERT's 95M. The 6M state encoder is 6.3%; please clarify which parameter count is meant.
  2. [Figure 3, §4.3] The spectrogram decoder used for the counterfactual visualizations is described only as 'separately trained' or 'freeze the encoder and train a spectrogram decoder'. Please provide architecture, training loss, and dataset details, or remove qualitative figure claims if this decoder is not fully documented.
  3. [§4.4, beat tracking] The beat-tracking experiment uses the ASAP dataset, but it is not clear whether the reported numbers follow the MAESTRO train/validation/test split or ASAP's own splits, and how the MAESTRO-trained encoder is adapted to ASAP segments. Please specify the experimental protocol precisely.
  4. [Figure 2 / Table 1] The term 'random' in Table 1 denotes samples drawn from the dataset, while Figure 2 uses 'random' for actions and states; please define each perturbation type in the caption or text. Also, temporal-shift win rates are averaged over all time indices; reporting the full distance dependence (as in Figure 2) is more informative.
  5. [§5] The conclusion appropriately states that scaling and generalization to settings without action annotations remain unclear. This is welcome, but the abstract's 'world model' and 'planning' claims should be tempered to match the actual evidence, especially given the points raised above.

Circularity Check

2 steps flagged

The 'transcription via planning' experiment is a supervised inverse fit, and the dynamics evaluation reuses the training loss; downstream MIR benchmarks remain independent.

specific steps
  1. fitted input called prediction [Section 3.4 (Eq. 5) and Section 4.5]
    "We therefore adopt an amortized optimization approach by training an inverse predictor a t+1 = h(s t, st+1, at) to approximate the solution [33, 34]."

    Eq. 5 defines planning as argmin over actions of the learned-dynamics losses ||f(s_t,a_{t+1}) - s_{t+1}||^2 + ||g(a_t)-a_{t+1}||^2. The implemented experiment never optimizes this objective: it trains h as a direct supervised map from state pairs and the previous action to the next ground-truth action, and decodes with a separately trained decoder. The transcription results in Table 3 are therefore outputs of a fitted inverse predictor, not of planning through f. The abstract's claim that the model enables 'piano transcription via planning, by searching for actions' reduces to a supervised fit on the ground-truth actions.

  2. self definitional [Section 4.3, Eq. (6) vs Section 3.3, Eq. (3)]
    "we measure the error against the ground truth s^gt_{t+1}: L(s_t, a_{t+1}, s^gt_{t+1}) = ||f(s_t, a_{t+1}) - s^gt_{t+1}||^2. (6) A well-formed dynamics model assigns lower error to correct state–action–target triplets and higher error when any component is perturbed."

    Eq. (6) is identical to the first term of the training objective Eq. (3), L(θ) = ||f(s_t, a_{t+1}) - s_{t+1}||^2 + λ||g(a_t) - a_{t+1}||^2. The section's evidence that the model 'captures action-conditioned temporal dynamics' is thus that the model scores well on the same MSE it was trained to minimize; the perturbation win rates and ΔL are computed from this same loss. At least part of the reported sensitivity is built into the objective rather than established by an independent counterfactual test.

full rationale

Music-JEPA's downstream MIR evaluations (beat tracking, composer identification, key recognition) are external benchmarks and provide independent evidence, so the paper is not globally circular. The strongest load-bearing problem is the 'transcription via planning' section: the planning objective in Eq. (5) is replaced by a supervised inverse model trained on ground-truth actions, so the planning claim reduces to a fit rather than a test of the learned dynamics. A second, milder circularity is that the dynamics evaluation in Section 4.3 reuses the training MSE (Eq. 6 = first term of Eq. 3) as the success metric, making the dynamics claim partly self-definitional. The Footnote 2 assumption that 'a_{t+1} produces s_{t+1}' is an unverified causal segmentation, and the conclusion acknowledges uncertain generalization; these are limitations rather than circular steps. Self-citations to prior JEPA work are standard and not used to forbid alternatives, so they do not count as circular. Overall, partial circularity in the planning and dynamics claims yields a score of 6.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claim rests on standard self-supervised learning assumptions plus several domain-specific choices: MAESTRO alignment, segment-local causality, EMA anti-collapse, and latent-MSE fidelity. There are no invented physical entities or new conserved quantities. The free parameters are hand-set hyperparameters that affect the learned dynamics and are not independently justified.

free parameters (5)
  • λ (action-prediction loss weight) = 0.5
    Hand-set in Eq. (3); controls balance between state and action dynamics; no ablation reported.
  • EMA momentum τ = 0.95
    Hand-set in Eq. (4); standard EMA value.
  • state/action segment length = 2 s
    Defines the discrete time step for the dynamics; hand-chosen.
  • patch sizes = state 25×15, action 25×6
    Hand-chosen; controls number of tokens Ks=256, Ka=240.
  • embedding dim and transformer depths = D=256; 12/8/6 layers
    Architecture choices without ablations; central to model capacity.
axioms (4)
  • domain assumption Pianoroll and pedal segments are time-aligned actions that generate the co-occurring audio segment (a_{t+1} produces s_{t+1}).
    Invoked in Eq. (1) and Footnote 2; load-bearing for causal and counterfactual claims.
  • domain assumption EMA teacher encoders and layer normalization prevent representation collapse.
    Eq. (4) and §3.3; standard but empirical, with no formal guarantee.
  • domain assumption MSE in latent space is a meaningful proxy for audio/action correctness.
    Eqs. (3) and (6) use latent MSE; dynamics evaluation and planning rely on this, and the decoder reconstruction is only partial validation.
  • domain assumption Offline paired data is sufficient to learn a world model without environment interaction.
    Abstract and §3; MAESTRO provides paired actions and states, but assumes no interactive exploration is needed.

pith-pipeline@v1.3.0-alltime-deepseek · 13724 in / 12806 out tokens · 133428 ms · 2026-08-01T06:05:17.200022+00:00 · methodology

0 comments
read the original abstract

Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 4 canonical work pages

  1. [1]

    # "!"#"! # $ MSE !̂!

    INTRODUCTION While humans may be passively exposed to a wide range of sound signals, understanding music typically arises from active engagement. For example, learning piano music often involves interacting with the instrument and under- standing how actions on the keyboard produce acoustic outcomes. Through such interaction, cognitive studies sug- gest t...

    Pith/arXiv arXiv 2026

  2. [2]

    RELATED WORKS Learning meaningful representations is a fundamental problem not only in music but more broadly in machine intelligence. While recent advances in generative model- ing have achieved strong performance, they often empha- size reproducing statistical patterns rather than capturing the underlying structure of how signals evolve [17]. This limit...

  3. [3]

    METHOD Music-JEPA models music as a dynamical system, where audio evolves over time under the influence of performance actions. We instantiate this framework on piano music us- ing paired audio and control sequences(x1:T , y1:T ), where xt is a short audio segment andy t the corresponding pi- anoroll and pedal signals. In our implementation, each segment ...

  4. [4]

    We first describe the dataset, training procedure (Section 4.1), and baselines (Section 4.2)

    EXPERIMENTS In this section, we evaluate Music-JEPA. We first describe the dataset, training procedure (Section 4.1), and baselines (Section 4.2). We then assess the model from three per- spectives: temporal dynamics (Section 4.3), representation quality on downstream MIR tasks (Section 4.4), and tran- scription via planning (Section 4.5). 4.1 Dataset and...

    arXiv

  5. [5]

    We demonstrate three key capabilities

    CONCLUSION AND FUTURE WORK We propose Music-JEPA, a music world model that learns action-conditioned temporal dynamics of audio in latent space. We demonstrate three key capabilities. First, the model captures temporal dynamics more effectively than audio-only JEPA baselines, showing consistent sensitivity to state–action transitions. Second, the learned ...

  6. [6]

    D. B. Huron,Sweet anticipation: Music and the psy- chology of expectation, 2006

    2006

  7. [7]

    Hohwy,The predictive mind

    J. Hohwy,The predictive mind. OUP Oxford, 2013

    2013

  8. [8]

    A path towards autonomous machine in- telligence version 0.9.2, 2022-06-27,

    Y . LeCun, “A path towards autonomous machine in- telligence version 0.9.2, 2022-06-27,” 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID: 251881108

    2022

  9. [9]

    Self-supervised learning from im- ages with a joint-embedding predictive architec- ture,

    M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y . LeCun, and N. Ballas, “Self-supervised learning from im- ages with a joint-embedding predictive architec- ture,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, V ancou- ver , BC, Canada, June 17-24, 2023. IEEE, 2023, pp. 15 619–15 629. [Online]. Avail...

    arXiv 2023

  10. [10]

    V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,

    M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. J. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V . Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma, S. Chandar, F. Meier, Y . LeCun, M. Rabbat, and N. Ballas, “V-JEPA 2: Self-supervi...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.09985 2025

  11. [11]

    A-JEPA: joint- embedding predictive architecture can listen,

    Z. Fei, M. Fan, and J. Huang, “A-JEPA: joint- embedding predictive architecture can listen,”CoRR, vol. abs/2311.15830, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2311.15830

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.15830 2023

  12. [12]

    Audio-jepa: Joint-embedding predictive archi- tecture for audio representation learning,

    L. Tuncay, E. Labbé, E. Benetos, and T. Pelle- grini, “Audio-jepa: Joint-embedding predictive archi- tecture for audio representation learning,”CoRR, vol. abs/2507.02915, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2507.02915

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.02915 2025

  13. [13]

    Investigating design choices in joint-embedding predictive architectures for general audio repre- sentation learning,

    A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “Investigating design choices in joint-embedding predictive architectures for general audio repre- sentation learning,” inIEEE International Con- ference on Acoustics, Speech, and Signal Pro- cessing, ICASSP 2024 - Workshops, Seoul, Re- public of Korea, April 14-19, 2024. IEEE, 2024, pp. 680–684. [Online]....

    arXiv 2024

  14. [14]

    Stem-jepa: A joint-embedding predictive architecture for musical stem compatibility estimation,

    A. Riou, S. Lattner, G. Hadjeres, M. Anslow, and G. Peeters, “Stem-jepa: A joint-embedding predictive architecture for musical stem compatibility estimation,” inProceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, B. Kaneshiro, G. J. Mysore, O...

    2024

  15. [15]

    Emergent musical properties of a transformer under contrastive self- supervised learning,

    Y . Kong, G. Meseguer-Brocal, V . Lostanlen, M. La- grange, and R. Hennequin, “Emergent musical properties of a transformer under contrastive self- supervised learning,” inProceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, South Korea, September 21-25, 2025, J. Nam, D. Jeong, K. Choi, L. Su, M. F...

    work page doi:10.5281/zenodo.17811358 2025

  16. [16]

    Contrastive learning of musical representations,

    J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” inProceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Srinivasamurthy, Eds., 2021, pp. 673–681. [Online]. Available: http...

    2021

  17. [17]

    Masked autoencoders that listen,

    P. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” inAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrav...

    2022

  18. [18]

    MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dannenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Z. Wang, Y . Guo, and J. Fu, “MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training,” in The Twelfth International Conference on Learning Representations, 2024. [Onlin...

    2024

  19. [19]

    Exponential moving av- erage normalization for self-supervised and semi- supervised learning,

    Z. Cai, A. Ravichandran, S. Maji, C. C. Fowlkes, Z. Tu, and S. Soatto, “Exponential moving av- erage normalization for self-supervised and semi- supervised learning,” inIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vi- sion Foundation / IEEE, 2021, pp. 194–203. [Online]. Available: https://opena...

    2021

  20. [20]

    Iterative amortized policy optimization,

    J. Marino, A. Piché, A. D. Ialongo, and Y . Yue, “Iterative amortized policy optimization,” inAd- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Pro- cessing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 15 667–...

    2021

  21. [21]

    High-resolution sustain pedal depth estimation from piano audio across room acoustics,

    H. Zhang, K. Fang, Z. Wang, and I. Fujinaga, “High-resolution sustain pedal depth estimation from piano audio across room acoustics,” inProceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, South Korea, September 21-25, 2025, J. Nam, D. Jeong, K. Choi, L. Su, M. Fuentes, T. Nakano, X. Hu, and H. H. ...

    work page doi:10.5281/zenodo.17811439 2025

  22. [22]

    Critiques of world models,

    E. P. Xing, M. Deng, J. Hou, and Z. Hu, “Critiques of world models,”CoRR, vol. abs/2507.05169,

    Pith/arXiv arXiv

  23. [23]

    Variance-covariance regularization improves representation learning,

    J. Zhu, R. Shwartz-Ziv, Y . Chen, and Y . Le- Cun, “Variance-covariance regularization improves representation learning,”CoRR, vol. abs/2306.13292,

    Pith/arXiv arXiv

  24. [24]

    PAN: A world model for general, interactable, and long-horizon world simulation,

    J. Xiang, Y . Gu, Z. Liu, Z. Feng, Q. Gao, Y . Hu, B. Huang, G. Liu, Y . Yang, K. Zhou, D. Abrahamyan, A. Ahmad, G. Bannur, J. Chen, K. Chen, M. Deng, R. Han, X. Huang, H. Kang, Z. Li, E. Ma, H. Ren, Y . Shinde, R. Shingre, R. Tanikella, K. Tao, D. Yang, X. Yu, C. Zeng, B. Zhou, H. Liu, Z. Hu, and E. P. Xing, “PAN: A world model for general, interactable,...

    doi:10.48550/arxiv.2511.09057 2025

  25. [25]

    Pointworld: Scaling 3d world models for in-the-wild robotic manipulation,

    W. Huang, Y . Chao, A. Mousavian, M. Liu, D. Fox, K. Mo, and L. Fei-Fei, “Pointworld: Scaling 3d world models for in-the-wild robotic manipulation,” CoRR, vol. abs/2601.03782, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2601.03782

    doi:10.48550/arxiv.2601.03782 2026

  26. [26]

    A theory of cortical responses,

    K. Friston, “A theory of cortical responses,”Philosoph- ical transactions of the Royal Society B: Biological sci- ences, vol. 360, no. 1456, pp. 815–836, 2005

    2005

  27. [27]

    C. L. Krumhansl,Cognitive F oundations of Musical Pitch. Oxford University Press, 2001

    2001

  28. [28]

    London,Hearing in Time: Psychological Aspects of Musical Meter

    J. London,Hearing in Time: Psychological Aspects of Musical Meter. Oxford University Press, 2012

    2012

  29. [29]

    Unsupervised disentanglement of content and style via variance- invariance constraints,

    Y . Wu, Z. Wang, B. Raj, and G. Xia, “Unsupervised disentanglement of content and style via variance- invariance constraints,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [Online]. Available: https://openreview.net/forum?id= Lut5t3qElA

    2025

  30. [30]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7,

    2021

  31. [31]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. Jé- gou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 2021, pp. 9630–9640. [Online]. Available: https://doi.org/10.1109/ICCV48922.2021.00951

    arXiv 2021

  32. [32]

    Lejepa: Provable and scalable self-supervised learning without the heuris- tics,

    R. Balestriero and Y . LeCun, “Lejepa: Provable and scalable self-supervised learning without the heuris- tics,”CoRR, vol. abs/2511.08544, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2511.08544

    Pith/arXiv arXiv doi:10.48550/arxiv.2511.08544 2025

  33. [33]

    Balancing information preservation and disentangle- ment in self-supervised music representation learn- ing,

    J. Wilkins, S. Ding, M. Fuentes, and J. P. Bello, “Balancing information preservation and disentangle- ment in self-supervised music representation learn- ing,” inIEEE Workshop on Applications of Sig- nal Processing to Audio and Acoustics, WASPAA 2025, Tahoe City, CA, USA, October 12-15, 2025. IEEE, 2025, pp. 1–5. [Online]. Available: https: //doi.org/10....

    arXiv 2025

  34. [34]

    Audio barlow twins: Self-supervised audio represen- tation learning,

    J. Anton, H. Coppock, P. Shukla, and B. W. Schuller, “Audio barlow twins: Self-supervised audio represen- tation learning,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  35. [35]

    PESTO: pitch estimation with self-supervised transposition-equivariant objective,

    A. Riou, S. Lattner, G. Hadjeres, and G. Peeters, “PESTO: pitch estimation with self-supervised transposition-equivariant objective,” inProceedings of the 24th International Society for Music Informa- tion Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, A. Sarti, F. Antonacci, M. San- dler, P. Bestagini, S. Dixon, B. Liang, G. Richard,...

    work page doi:10.5281/zenodo.10265343 2023

  36. [36]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Y . Bengio and Y . LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv 2015

  37. [37]

    Bootstrap your own latent - a new approach to self-supervised learning,

    J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent - a new approach to self-supervised learning,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan,...

    2020

  38. [38]

    ASAP: a dataset of aligned scores and performances for piano transcription,

    F. Foscarin, A. McLeod, P. Rigaux, F. Jacquemard, and M. Sakai, “ASAP: a dataset of aligned scores and performances for piano transcription,” inProceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, Montreal, Canada, October 11-16, 2020, J. Cumming, J. H. Lee, B. McFee, M. Schedl, J. Devaney, C. McKay, E. Zan...

    2020

  39. [39]

    Closing the train-test gap in world models for gradient-based planning,

    A. Parthasarathy, N. Kalra, R. Agrawal, Y . LeCun, O. Bounou, P. Izmailov, and M. Goldblum, “Closing the train-test gap in world models for gradient-based planning,”CoRR, vol. abs/2512.09929, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2512.09929

    doi:10.48550/arxiv.2512.09929 2025

  40. [40]

    Tem- poral straightening for latent planning,

    Y . Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. J. Rudner, Y . LeCun, and M. Ren, “Tem- poral straightening for latent planning,”CoRR, vol. abs/2603.12231, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2603.12231

    doi:10.48550/arxiv.2603.12231 2026

  41. [41]

    Tutorial on amortized optimization for learning to optimize over continuous domains,

    B. Amos, “Tutorial on amortized optimization for learning to optimize over continuous domains,” CoRR, vol. abs/2202.00665, 2022. [Online]. Available: https://arxiv.org/abs/2202.00665

    arXiv 2022

  42. [42]

    Latent geometry beyond search: Amortizing planning in world models,

    H. Nguyen, X. Xu, and X. Huang, “Latent geometry beyond search: Amortizing planning in world models,” 2026. [Online]. Available: https://arxiv.org/ abs/2605.08732

    Pith/arXiv arXiv 2026

  43. [43]

    Enabling factorized piano music modeling and generation with the MAE- STRO dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Si- mon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAE- STRO dataset,” inInternational Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/forum?id=r1lYRjC9F7

    2019

  44. [48]

    Beat this! accurate beat tracking without DBN postprocessing,

    F. Foscarin, J. Schlüter, and G. Widmer, “Beat this! accurate beat tracking without DBN postprocessing,” inProceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, B. Kaneshiro, G. J. Mysore, O. Nieto, C. Donahue, C. A. Huang, J. H. Lee, B. McFee,...

    work page doi:10.5281/zenodo.14877491 2024

  45. [49]

    madmom: a new Python Audio and Mu- sic Signal Processing Library,

    S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer, “madmom: a new Python Audio and Mu- sic Signal Processing Library,” inProceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands, 10 2016, pp. 1174– 1178

    2016

  46. [50]

    High-resolution piano transcription with pedals by regressing onset and offset times,

    Q. Kong, B. Li, X. Song, Y . Wan, and Y . Wang, “High-resolution piano transcription with pedals by regressing onset and offset times,”IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3707–3717,

  47. [51]

    Available: https://doi.org/10.1109/ TASLP.2021.3121991

    [Online]. Available: https://doi.org/10.1109/ TASLP.2021.3121991

    arXiv 2021

  48. [52]

    Scoring time intervals using non-hierarchical transformer for automatic piano transcription,

    Y . Yan and Z. Duan, “Scoring time intervals using non-hierarchical transformer for automatic piano transcription,” inProceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, B. Kaneshiro, G. J. Mysore, O. Nieto, C. Donahue, C. A. Huang, J. H. Lee...

    doi:10.5281/zenodo.14877493 2024

  49. [541]

    Available: http://archives.ismir.net/ ismir2020/paper/000127.pdf

    [Online]. Available: http://archives.ismir.net/ ismir2020/paper/000127.pdf

  50. [2021]

    [Online]

    OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy

    2021

  51. [2023]

    Available: https://doi.org/10.48550/ arXiv.2306.13292

    [Online]. Available: https://doi.org/10.48550/ arXiv.2306.13292

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.13292

  52. [2025]

    Available: https://doi.org/10.48550/ arXiv.2507.05169

    [Online]. Available: https://doi.org/10.48550/ arXiv.2507.05169

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.05169