Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-supervised encoder trained on MIDI piano-roll images organizes musical knowledge by timescale: phrases at coarse levels, notes and harmony at fine levels, and the same frozen encoder drives generation and inpainting.

desk verdict A practical, CPU-runnable music representation and generation system whose headline hierarchy claim needs a random-init control before 'emergent' is supported. read the letter →

arxiv 2608.04378 v1 pith:VXT5XMDH submitted 2026-08-05 cs.SD cs.LGeess.AS

Scott H. Hawley This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords self-supervisedlearningsymbolicmusicworldmodelJEPAhierarchicalrepresentationMIDIpiano-rollflowmatchinginpainting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single self-supervised encoder trained on MIDI piano-roll images, with no labels and no music-theory vocabulary, spontaneously organizes musical knowledge by timescale: phrase boundaries become readable at its coarsest levels, while note density and harmonic detail become readable at its finest. It also claims that the same frozen encoder can drive controllable generation, reproducing a target window almost exactly and performing masked inpainting through per-level conditioning dropout. A sympathetic reader would care because it suggests that a musical hierarchy can be discovered, not taught, and that one representation can serve both understanding and generation, which is what a collaborative music agent needs.

What carries the argument

The central object is a 2.55M-parameter hierarchical Swin V2 transformer encoder trained on 128x128 binary piano-roll crops with a combined JEPA-style objective: an equivariance loss that makes embedding distance track pitch/time shift magnitude, a distributional regularizer (SIGReg) that prevents collapse, a masked-embedding-prediction loss, and a soft factorization loss that constrains the geometry of pitch versus time differences. The hierarchy of six levels (L0 finest to L5 coarsest) is what carries the argument: probing each level separately shows which musical properties live where. For generation, the same encoder's PCA-reduced embeddings condition a flow-matching model that samples in pixel space, with spatial dropout on the conditioning enabling inpainting.

What would settle it

Retrain the encoder with several different random seeds and run the phrase-boundary and note-density probes at every hierarchy level: if the best level for phrase boundaries moves from the coarse end to the fine end across seeds, or if per-level scores overlap within seed-to-seed noise, the central hierarchy claim would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the level at which a musical property becomes decodable from the frozen embeddings tracks the property's musical time scale. Probes placed on different levels of the hierarchical Swin V2 encoder find phrase boundaries at L0–L2, the coarse end, and note density and harmonic content at L4–L5, the fine end. Temporal and phrase structure emerge from the self-supervised objectives alone, whereas harmonic content must be asked for; adding a small chord-supervision head raises joint chord recovery from .18 to .54 and key detection from .16 to .70 even though key was never supervised. On the generation side, a conditional flow-matching model conditioned on PCA-reduced embeddings reproduces a target window at pixel F1 0.996, and the same per-level dropout used to control variation also enables masked inpainting with no inpainting-specific sampler.

Load-bearing premise

The load-bearing premise is that the probe scores in Table 1 are stable enough that the level with the best score truly identifies where a property lives; the table selects each probe's best level after training across six levels, with no error bars or seed variation for most probes, so if those peaks shift with random seeds or probe hyperparameters, the timescale correspondence is not a robust property of the representation.

Editorial extensions

If this is right

  • If the hierarchy claim holds, phrase structure and temporal organization can be read directly from an unsupervised representation, so downstream music-understanding systems may not need hand-labeled phrase data for those tasks.
  • Harmonic content, by contrast, is not emergent: the paper's chord-supervision results imply that harmonic understanding requires a small supervised head, though that head also improves never-supervised key detection.
  • Because one frozen encoder conditions both reconstruction and inpainting, a single perceptual front end can serve a co-creation agent's understanding and generation without task-specific fine-tuning.
  • The CPU-friendly runtime (2.8 seconds for a suggestion) means this kind of agent can run locally, keeping works-in-progress off cloud services.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The level-to-timescale correspondence may be a general property of hierarchical self-supervised models on structured images, so the same probing recipe could be applied to spectrogram-like or audio inputs to test whether the mapping transfers across domains.
  • The success of PCA-reduced conditioning suggests the encoder's many dimensions per patch are redundant for generation, so a lighter conditioning pathway might suffice and could further cut latency.
  • The chord-supervision boost to key detection hints that weakly correlated labels can pull latent geometry into line for properties that were never supervised; this might be testable for other correlated musical attributes such as mode or meter.
  • The paper leaves implicit that the same level-dropout mechanism could be used for style transfer by dropping coarse levels while keeping fine ones, rewriting harmony while preserving notes; that is a direct extension the paper does not claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents MRJS, a 2.55M-parameter hierarchical Swin V2 encoder trained on MIDI piano-roll images with a JEPA-style objective combining shift equivariance, SIGReg regularization, masked embedding prediction, and a soft factorization loss. The authors probe frozen per-level embeddings with a suite called STORMBIRD, reporting that phrase boundaries are decodable at coarse levels while note density and harmonic content are decodable at fine levels, and that chord supervision improves chord and key detection while phrase structure arises without supervision. They then attach a PCA-reduced conditional flow-matching decoder, showing near-perfect reconstruction of a target window (pixel F1 0.996) and using per-level conditioning dropout for masked inpainting, with CPU-friendly latency. The paper positions this as the "ears" and "mouth" of a collaborative music co-creation agent.

Significance. If the level-to-timescale hierarchy claim holds, it is a notable result: a label-free self-supervised model would discover a musical organization (phrases at coarse scales, notes at fine scales) that matches music-theoretic intuition. The work is also practically motivated and unusually transparent: the appendices report per-level tables for all probes, independent external annotations (POP909, EMOPIA, Dai et al.) are used, and a live demo and timing data are provided. The methodological innovation of using a single frozen hierarchical encoder for both understanding and controllable generation is appealing and aligns with the representation-autoencoder paradigm. However, the central interpretative claim is currently vulnerable to an architectural confound, as detailed below, and the generation evaluation is thin, so the significance hinges on the additional controls the revision should add.

major comments (4)
  1. [§3, Table 1 and Figure 3] The central claim that "the level at which a musical property becomes decodable tracks its musical time scale" is not controlled against the architecture's built-in receptive-field hierarchy. In the Swin V2 encoder, Level 0 embeddings are global (1x1 patch) and Level 5 embeddings are local (4x4 patches). A linear probe on coarse global features will naturally be better for global properties such as phrase boundaries, and a probe on fine local features will naturally be better for local statistics such as note density, regardless of whether the self-supervised training has learned any musical structure. To support the emergence claim, the authors must add a random-initialized (untrained) encoder as a control and show that the coarse/fine correspondence is absent or substantially weaker before training. Without this control, the abstract's statement that temporal and phrase structure "emerge from the self-supervised objectives alone" is not established.
  2. [Table 1 and Appendix B] The consolidated table reports each probe at the hierarchy level where it scores best, with the winning level selected after training across six levels, and no error bars or seed variation for most probes. The per-level appendix tables (A2–A16) mitigate selection bias somewhat, but the peak level could shift with random seeds, probe hyperparameters, or the choice of probe suite. The authors should report means and standard deviations over at least a few encoder training seeds, or otherwise demonstrate that the winning levels are stable. This is load-bearing because the level-to-timescale correspondence is the paper's main interpretative result.
  3. [§3, Table 1, key detection row] The improvement in key detection from 0.16 to 0.70 under chord supervision is attributed to "never supervised" transfer, but chord labels are strongly correlated with key in tonal music. A model trained with chord supervision may trivially infer key from the chord progression. To support the claimed emergence, the authors should add a control with shuffled or randomized chord labels (or otherwise remove the statistical association) and test whether key detection still improves. This is a concrete, fixable test that would distinguish genuine representation-level transfer from label correlation.
  4. [§4, Table 2(a)] The pixel F1 of 0.996 is a reconstruction number: the flow is conditioned on the input window's own PCA-reduced embeddings and must reproduce that same window. This is a useful sanity check for the RAE paradigm, but it is not a generation-quality metric. The more interesting claim is that the same pipeline produces usable variations and inpaintings, yet the quantitative evaluation of generation is limited to note-density restoration over 6 songs x 3 seeds (Table 2(b)) and a few qualitative examples in Figure 4. The paper should add objective or perceptual evaluation of the generated variations (e.g., diversity, musical plausibility, or a listening study) before claiming that the system provides controllable generation suitable for co-creation.
minor comments (6)
  1. [Footnote 1] There is a typo in "upplemental Materials" (missing 'S').
  2. [Figure 3] The caption lists levels as "L54x4patches L48x8patches L316x16patches L232x32patches L164x64patches L0128x128patch" which reads as L5 first and L0 last; this is the reverse of the natural scale order and made the figure harder to parse. Please reorder from L0 to L5.
  3. [Table 1 caption] The caption says "cells are shaded per row with the plasma colormap (yellow = best)" but plasma generally maps yellow to high values; this is fine, but the caption should also state whether the bar heights are proportional to receptive field as indicated, and the reader has no way to verify the shaded values from the text version. A note that the interactive version has the full per-level tables would help.
  4. [Appendix B, Table A1] The roster labels like "c55enc3_vkrDxf" and "chall-scr1" are opaque; a one-line description in the table itself would improve readability, even though the main text explains some of them.
  5. [§2, Equation (2)] The notation \hat{\delta} is defined only after the equation; moving the definition before the equation would help. Also, the per-axis division by (\Delta t_max, \Delta p_max) means \hat{\delta} components lie in [0,1] only for nonnegative shifts; if negative shifts are used, the text should state that the absolute value or symmetric range is intended.
  6. [§4, Table 2(b)] The row "fine levels (L4–L5) @0.85" shows 96% restoration but the text says "Restoration falls monotonically with dropout strength;" the monotonicity is evident for the three rows, but the table lacks a "no dropout" row for the masked scenario; please clarify whether the 100% row is the no-dropout condition or the unmasked reconstruction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central probing and generation claims are empirical and transparently conditioned, with no derivation reducing to its own input.

full rationale

The paper's central claim, that probe performance peaks at hierarchy levels matching musical time scale, rests on linear probes evaluated against external human annotations (POP909 chord labels, Dai et al. phrase boundaries). These labels are not used in the self-supervised objective, so the coarse/fine peak pattern is not entailed by the loss. The equivariance diagnostics in Appendix C measure the same quantity as the Lequiv training loss, but the main text does not rely on them as evidence of emergent structure; the temporal-offset probe is explicitly weak (R2 .22-.26) and the paper treats that weakness as expected from the near-stationarity of piano rolls. The F1=0.996 generation figure is a reconstruction from the target window's own PCA-reduced embeddings, and the table caption states this conditioning explicitly ('Conditioned on a real window's own embeddings, the flow reproduces it nearly pixel-perfectly with no post-hoc alignment: the PCA-reduced conditioning retains enough to rebuild the input'), so it is not a disguised predictive claim. Self-citations to the author's prior MIDI-RAE-JEPA report provide architectural context and are not used as an external uniqueness argument or to forbid alternative approaches. Overall the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The framework's contributions rest on a small number of hand-selected hyperparameters (loss weights, shift ranges, dropout schedules, PCA threshold) and on domain assumptions about piano-roll sufficiency and linear-probe interpretability. No new physical or latent entities are postulated; the hierarchy levels are standard Swin scales.

free parameters (9)
  • alpha (equivariance distance scale) = not stated (ablation-selected)
    Scales target embedding distance in Lequiv; chosen by hand or ablation per Appendix A.
  • lambda (equivariance vs SIGReg weight) = reference value 0.15 scalar on L0 to L3
    Weights the equivariance and SIGReg losses; Appendix E reports per-level SIGReg strength variants.
  • lambda_MEP and lambda_fact = not stated
    Weights for masked embedding prediction and soft factorization losses; inherited from earlier report [17].
  • shift ranges (deltat_max, deltap_max) = 48 time pixels, 12 pitch semitones
    Max shift magnitudes for view generation; chosen by hand.
  • EMA momentum eta = 0.96
    Teacher update momentum; standard DINO and I-JEPA choice.
  • crop size and quantization = 128x128 binary pixels, 32nd-note grid
    Fixed by CPU runtime constraint; 32nd notes chosen so a blank frame separates repeated notes.
  • PCA variance retention = 90% per level, 4,835 floats total
    Reduces conditioning dimensionality; threshold chosen to retain most variance.
  • conditioning dropout probabilities = e.g., L4-L5 at 0.85, L4-L5 at 1.0, all levels at 1.0
    Controls variation distance and inpainting; chosen per demo and Table 2.
  • chord supervision weight lambda_chord = 0.5 for production +chords model
    Weight for auxiliary chord heads; ablations show 0.5 and 1.0 work best, 0.1 is poor.
assumptions (6)
  • domain assumption Binary piano-roll images on a 32nd-note grid are a sufficient representation for songwriting co-creation.
    Velocity and audio detail are discarded; justified by the use case in Section 2 and the audio-to-MIDI upstream assumption.
  • domain assumption Linear probes on frozen embeddings measure what the model perceives.
    Standard practice, but assumes linear decodability reflects representation content; Section 3.
  • standard math SIGReg Epps-Pulley test enforces an isotropic Gaussian prior and prevents representational collapse.
    Relies on LeJEPA theory [2] as cited in Section 2.
  • domain assumption Flow matching with 10 Euler steps at guidance 1.0 yields high-quality samples.
    Empirically validated only on 6 songs by 3 seeds; the general claim rests on this small evaluation.
  • domain assumption EMA teacher with eta=0.96 provides stable regression targets for masked embedding prediction.
    Standard heuristic from DINO and I-JEPA, used in Appendix A.
  • domain assumption The STORMBIRD probe suite covers the musically relevant properties for the hierarchy claim.
    The choice of probes (density, chroma, chords, key, phrase, time, cross-song) is the measurement lens for the central claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation." pith.science (2026). https://pith.science/paper/VXT5XMDH

@misc{pith2026260804378,
  author       = {Pith},
  title        = {Pith review of: Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VXT5XMDH}},
  note         = {Machine review of arXiv:2608.04378}
}
abstract

Collaborative music agents need internal representations rich enough to support both understanding and generation, yet flexible enough for a workflow where the human retains agency. We present a hierarchical self-supervised ``world model'' for symbolic music: a 2.55M-parameter Swin V2 encoder trained on MIDI piano-roll images with JEPA-style objectives (pitch- and time-shift equivariance, masked embedding prediction, and a distributional regularizer), using no labels and no music-theory vocabulary. Probing the frozen embeddings shows that the level at which a musical property becomes decodable tracks its musical time scale: phrase boundaries are read off the coarsest levels, note density and harmonic detail off the finest. Temporal and phrase structure emerge from the self-supervised objectives alone, while harmonic content must be asked for; a small chord-supervision head raises joint chord recovery from .18 to .54, and key detection, which is never supervised, from .16 to .70. Following the Representation AutoEncoder paradigm, a conditional flow-matching model stands in for a trained decoder, flowing in pixel space from PCA-reduced conditioning: it reproduces a target window at pixel F1 $0.996$, and the same per-level conditioning dropout that controls how far variations stray also enables graphical prompting for masked inpainting with no inpainting-specific sampler. The pipeline runs on CPU producing a suggestion in $2.8$ s, or $0.6$ s on Apple MPS, which we demonstrate in a live interactive demo. In concert with an LLM-based brain, these capabilities supply the core of a collaborative music creation agent in service of, rather than in place of, human agency.

Figures

Figures reproduced from arXiv: 2608.04378 by the authors.

Figure 1
Figure 1. The Ears/Brain/Mouth workflow for a conversational music co-writing agent. This paper [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Just as a vision world model pans and tilts a camera to scan a panoramic scene, we take [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our 2.55 million-parameter hierarchical Swin V2 Encoder. Model architecture hyperparam [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Unified generation and inpainting. The encoder’s PCA-reduced conditioning maps pass [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Soft factorization targets. Differences between pairs of embeddings are encouraged to be [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Example soft factorization results, Level 0. Left: PCA of normalized embedding difference [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Soft factorization results for levels L1, L2, and L3 (top to bottom), in the format of Figure 6. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Soft factorization results for L4 and L5 (top to bottom). The factorization loss is applied [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Per-level embedding distance versus pitch transposition shift for [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Per-level embedding distance versus pitch transposition shift for [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Per-level embedding distance versus pitch transposition shift for [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Per-level embedding distance versus time translation shift for [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Per-level embedding distance versus time translation shift for [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Per-level embedding distance versus time translation shift for [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

  1. [1]

    Self-supervised learning from images with a joint- embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. InConference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  2. [2]

    LeJEPA: Provable and scalable self-supervised learning without the heuristics, 2025

    Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics, 2025

    work page 2025

  3. [3]

    Bittner, Juan José Bosch, David Rubinstein, Gabriel Meseguer-Brocal, and Sebastian Ewert

    Rachel M. Bittner, Juan José Bosch, David Rubinstein, Gabriel Meseguer-Brocal, and Sebastian Ewert. A lightweight instrument-agnostic model for polyphonic note transcription and multipitch estimation. InProceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Singapore, 2022

    work page 2022

  4. [4]

    Zapata, and Xavier Serra

    Dmitry Bogdanov, Nicolas Wack, Emilia Gómez, Sankalp Gulati, Perfecto Herrera, Oscar Mayor, Gerard Roma, Justin Salamon, José R. Zapata, and Xavier Serra. Essentia: An audio analysis library for music information retrieval. InProceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR), pages 493–498, Curitiba, Brazil, 2013

    work page 2013

  5. [5]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InInternational Conference on Computer Vision, 2021

    work page 2021

  6. [6]

    Codified audio language modeling learns useful representations for music information retrieval

    Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful representations for music information retrieval. InProceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2021

    work page 2021

  7. [7]

    Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm

    Ke Chen, Gus Xia, and Shlomo Dubnov. Music SketchNet: Controllable music generation via factorized representations of pitch and rhythm.arXiv preprint arXiv:2008.01291, 2020

    work page Pith review arXiv 2008

  8. [8]

    Pixelflow: Pixel-space generative models with flow, 2025

    Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo. Pixelflow: Pixel-space generative models with flow, 2025

    2025

Show all 45 references
  1. [9]

    Dannenberg

    Shuqi Dai, Huan Zhang, and Roger B. Dannenberg. Automatic analysis and influence of hierarchical structure on melody, rhythm and harmony in popular music. InProceedings of the Joint Conference on AI Music Creativity (AIMC), 2020. arXiv:2010.07518. Human- verified phrase-level ...

    2020 arXiv

  2. [10]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021

    2021

  3. [11]

    Generative modelling in latent space, 2025

    Sander Dieleman. Generative modelling in latent space, 2025

    2025

  4. [12]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. InProceedings of the International Conference on Machine Learning (ICML), 2024

    2024

  5. [13]

    Learning and leveraging world models in visual representation learning.arXiv preprint arXiv:2403.00504, 2024

    Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning.arXiv preprint arXiv:2403.00504, 2024

    2024 arXiv

  6. [14]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  7. [15]

    Using a joint-embedding predictive architecture for symbolic music understanding

    Rafik Hachana and Bader Rasheed. Using a joint-embedding predictive architecture for symbolic music understanding. InNeurIPS 2025 Workshop on AI for Music, 2025. 7

    2025

  8. [16]

    Scott H. Hawley. Pictures of midi: Controlled music generation via graphical prompts for image-based diffusion inpainting, 2024

    2024

  9. [17]

    Scott H. Hawley. MIDI-RAE-JEPA: Hierarchical representation learning and generation for symbolic music, 2026

    2026

  10. [18]

    EMOPIA: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation

    Hsiao-Tzu Hung, Joann Ching, Seungheon Doh, Nabin Kim, Juhan Nam, and Yi-Hsuan Yang. EMOPIA: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation. InProc. Int. Society for Music Information Retrieval Conf., 2021

    2021

  11. [19]

    How far can pretrained LLMs go in symbolic music? controlled comparisons of supervised and preference- based adaptation.arXiv preprint arXiv:2601.22764, 2026

    Deepak Kumar, Emmanouil Karystinaios, Gerhard Widmer, and Markus Schedl. How far can pretrained LLMs go in symbolic music? controlled comparisons of supervised and preference- based adaptation.arXiv preprint arXiv:2601.22764, 2026

    2026

  12. [20]

    A path towards autonomous machine intelligence.OpenReview preprint, 2022

    Yann LeCun. A path towards autonomous machine intelligence.OpenReview preprint, 2022. Version 0.9.2

    2022

  13. [21]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  14. [22]

    Swin transformer V2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer V2: Scaling up capacity and resolution. InConference on Computer Vision and Pattern Recognition, 2022

    2022

  15. [23]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InInternational Conference on Computer Vision, 2021

    2021

  16. [24]

    One-step latent-free image generation with pixel mean flows, 2026

    Yiyang Lu, Susie Lu, Qiao Sun, Hanhong Zhao, Zhicheng Jiang, Xianbang Wang, Tianhong Li, Zhengyang Geng, and Kaiming He. One-step latent-free image generation with pixel mean flows, 2026

    2026

  17. [25]

    CMI-Bench: A comprehensive benchmark for evaluating music instruction following

    Yinghao Ma, Siyou Li, Juntao Yu, Emmanouil Benetos, and Akira Maezawa. CMI-Bench: A comprehensive benchmark for evaluating music instruction following. InProceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR), 2025

    2025

  18. [26]

    Pnp-flow: Plug-and-play image restoration with flow matching

    Ségolène Martin, Anne Gagneux, Paul Hagemann, and Gabriele Steidl. Pnp-flow: Plug-and-play image restoration with flow matching. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Representation Learning, volume 2025, pages 45466–45492, 2025

    2025

  19. [27]

    Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in Python. InProceedings of the 14th Python in Science Conference (SciPy), pages 18–24, 2015

    2015

  20. [28]

    Polyffusion: A diffusion model for polyphonic score generation with internal and external controls

    Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: A diffusion model for polyphonic score generation with internal and external controls. InProceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR), 2023

    2023

  21. [29]

    DINOv2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2024

    2024 arXiv

  22. [30]

    Muckley, Ricky T

    Ashwini Pokle, Matthew J. Muckley, Ricky T. Q. Chen, and Brian Karrer. Training-free linear image inverses via flows.Transactions on Machine Learning Research, 2024

    2024

  23. [31]

    PhD thesis, Columbia University, 2016

    Colin Raffel.Learning-Based Methods for Comparing Sequences, with Applications to Audio- to-MIDI Alignment and Matching. PhD thesis, Columbia University, 2016

    2016

  24. [32]

    Stem- JEPA: A joint-embedding predictive architecture for musical stem compatibility estimation

    Alain Riou, Stefan Lattner, Gaëtan Hadjeres, Michael Anslow, and Geoffroy Peeters. Stem- JEPA: A joint-embedding predictive architecture for musical stem compatibility estimation. In International Society for Music Information Retrieval Conference, 2024

    2024

  25. [33]

    Crash: Raw audio score-based generative modeling for controllable high-resolution drum sound synthesis, 2021

    Simon Rouard and Gaëtan Hadjeres. Crash: Raw audio score-based generative modeling for controllable high-resolution drum sound synthesis, 2021. 8

    2021

  26. [34]

    Muscriptor: An open model for multi-instrument music transcription, 2026

    Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, and Alexandre Défossez. Muscriptor: An open model for multi-instrument music transcription, 2026

    2026

  27. [35]

    Rick Rubin: The 60 minutes interview.60 Minutes, CBS News

    Rick Rubin. Rick Rubin: The 60 minutes interview.60 Minutes, CBS News. Interview by Anderson Cooper, January 2023

    2023

  28. [36]

    Chih-Pin Tan, Alvin W. Y . Su, and Yi-Hsuan Yang. Melody infilling with user-provided structural context. InProceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR), Bengaluru, India, 2022

    2022

  29. [37]

    Improving and generalizing flow-based genera- tive models with minibatch optimal transport.Transactions on Machine Learning Research,

    Alexander Tong, Kilian FATRAS, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based genera- tive models with minibatch optimal transport.Transactions on Machine Learning Research,

  30. [38]

    Music-JEPA: Learning a world model of sound from action, 2026

    Ziyu Wang, Kun Fang, and Yann LeCun. Music-JEPA: Learning a world model of sound from action, 2026

    2026

  31. [39]

    Visreg: Variance-invariance-sketching regularization for jepa training, 2026

    Haiyu Wu, Randall Balestriero, and Morgan Levine. Visreg: Variance-invariance-sketching regularization for jepa training, 2026

    2026

  32. [40]

    MelodyGLM: Multi-task pre-training for symbolic melody generation.arXiv preprint arXiv:2309.10738, 2023

    Xiao Wu, Zihao Huang, Kai Zhang, Jun Yu, Xu Tan, Tao Zhang, Yan Li, Zhan Wang, and Lingling Sun. MelodyGLM: Multi-task pre-training for symbolic melody generation.arXiv preprint arXiv:2309.10738, 2023

    2023 arXiv

  33. [41]

    MIDI-LLaMA: An instruction-following multimodal LLM for symbolic music understanding.arXiv preprint arXiv:2601.21740, 2026

    Meng Yang, Jon McCormack, Maria Teresa Llano, Wanchao Su, and Chao Lei. MIDI-LLaMA: An instruction-following multimodal LLM for symbolic music understanding.arXiv preprint arXiv:2601.21740, 2026

    2026

  34. [42]

    ARIMA: Reconstruction-grounded predictive representa- tion learning for symbolic music.arXiv preprint arXiv:2607.10003, 2026

    Mingyang Yao and Zhaoxiang Feng. ARIMA: Reconstruction-grounded predictive representa- tion learning for symbolic music.arXiv preprint arXiv:2607.10003, 2026

    2026 arXiv

  35. [43]

    ABC-Eval: Benchmarking large language models on symbolic music understanding and instruction following.arXiv preprint arXiv:2509.23350, 2025

    Jiahao Zhao, Yunjia Li, Wei Li, and Kazuyoshi Yoshii. ABC-Eval: Benchmarking large language models on symbolic music understanding and instruction following.arXiv preprint arXiv:2509.23350, 2025

    2025

  36. [44]

    /all”; values = best across levels, one consistent probe-suite vintage; “/all

    Bowen Zheng, Nan Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders.arXiv preprint arXiv:2510.11690, 2025. 9 Supplemental Materials / Appendices The following appendices archive the complete measurement record behind the paper. An inte...

    2025 arXiv

  37. [2024]

    Expert Certification

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.