REVIEW 3 major objections 6 minor 24 references
Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatise
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A chain of image captioning, text-to-music diffusion, and latent-overlap stitching can turn the abstract pages of Cardew's Treatise into a coherent improvised audio piece.
desk verdict A short creative-systems paper that assembles known parts into a concrete, audible pipeline for Cardew's Treatise, but overclaims 'seamless and cohesive' without validating the one new stitching rule. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the outpainting latent continuation rule, $\epsilon' = \operatorname{concat}(z_k[:, T/2 :, :], \epsilon[:, : T/2, :])$, which replaces the fresh Gaussian noise input of MusicLDM's latent diffusion model with the second half of the previously generated latent plus new noise. This makes the model generate a continuation of its own prior output, so segment boundaries are handled inside the generative process rather than in post-processing. Around it stands the rest of the pipeline: ChatGPT 4o converts each score image into four descriptive sentences, prefix keywords such as 'sine wave is playing' set the style, CLAP embeds the prompt into the text-audio conditioning space, and the vocoder decodes the denoised latents into the final waveform.
What would settle it
Record the published tracks and compute the log-spectral distance across each ten-second segment boundary; if the outpainting overlap produces no smaller discontinuity than cutting and crossfading two independently generated segments, then the 'seamless continuation' claim is not supported. Alternatively, ask listeners to identify stitch points in the outpainting version versus a naive-concatenation baseline; chance-level performance would falsify the smoothness claim.
Extended reading notes
Core claim
The authors establish that a pipeline of image to text to audio—ChatGPT 4o interpreting the score pages, CLAP embedding the resulting prompts into a shared text-audio space, and MusicLDM generating the audio from those embeddings—can render pages 1 through 33 of Treatise as a continuous improvised piece. The key technical move is outpainting: instead of generating each segment independently and blending afterward, the second half of the previous latent output, $z_k[:, T/2 :, :]$, is concatenated with fresh Gaussian noise to form the input $\epsilon'$ for the next diffusion step, so the model itself is asked to continue. The paper reports that this produces smooth, score-following compositions whose complexity rises, falls, and rises again with the pages, and that page 33 provides a cadence-like ending. Three realizations, distinguished only by prefix keywords (sine wave, string orchestra, experimental), are offered as evidence of the method's flexibility.
Load-bearing premise
The seamlessness claim rests on the unvalidated assumption that feeding the diffusion model the second half of a previous latent plus fresh noise yields a valid continuation, rather than pushing the model out of its training distribution and producing artifacts.
Editorial extensions
If this is right
- Any graphic score that can be described in words becomes generatable audio without manually transcribing musical meaning; the same chain could point at other visual scores or even at paintings.
- The outpainting latent-overlap rule is a drop-in stitching method for MusicLDM-style generators, letting long-form pieces be assembled from discrete prompts while the model smooths the joins.
- The prefix keywords act as a performative control: the same 33 pages yield a sine-wave electronic realization, a string-orchestra realization, and an experimental realization, so the interpreter's role shifts into choosing textual style constraints.
- Page 33's cadence-like ending suggests that the visual structure of the source score can impose global form on the generated audio, not just local texture.
Reading between the lines
- A direct image-to-audio latent mapping, such as aligning CLIP with CLAP as the authors say they plan, might outperform the ChatGPT text bottleneck because captioning discards visual detail; comparing caption-driven versus CLIP-driven realizations would test whether the text stage is a feature or a loss.
- The outpainting rule could be validated by measuring whether the model's latent stays on the training manifold; if it drifts, the perceived smoothness may come from the decoder or from simple overlap-averaging rather than from true continuation.
- The method's reliance on a fixed ten-second window means longer-range structure across pages arises only through the one-half overlap; a multi-scale overlap or hierarchical prompting would be a natural extension to control large-scale form.
- Applying the same pipeline to pages 34 through 193, which were excluded for length, would test whether the reported global arc (rise, fall, rise, cadence) is a property of the score or an artifact of the chosen 33-page excerpt.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes an AI-based method for turning pages of Cornelius Cardew's Treatise into music. The pipeline uses ChatGPT-4o to convert each selected page into four textual prompts, prefixes the prompts with style keywords, embeds the prompts with CLAP, and feeds the embeddings to MusicLDM, a pretrained latent diffusion text-to-music model. To join consecutive 10-second segments, the authors modify the initial noise for each subsequent segment according to Eq. (1): the second half of the previous latent output is concatenated with fresh Gaussian noise. They call this 'outpainting' and claim it produces seamless, cohesive compositions that follow the score. The paper includes audio examples and qualitative descriptions in Appendix B but no quantitative evaluation or ablation.
Significance. If the method works as claimed, the paper would offer a creative application connecting graphic scores, LLM-based captioning, and text-to-music diffusion, and the audio examples would be valuable for the community. The approach is concrete and built on publicly available components, and the paper states the link to the generated pieces. However, the main claims are not empirically substantiated: the 'follows the score' and 'seamless' assertions rest on subjective self-assessments, and the novel outpainting step in Eq. (1) is not validated or compared with simpler alternatives. The contribution is therefore better characterized as a proposed system with promising demonstrations than as a demonstrated method with verified properties.
major comments (3)
- [III and Appendix B] The central claim that the system produces 'seamless and cohesive compositions' that 'follow the score' is supported only by the authors' own listening descriptions in Appendix B. There is no listening study with independent raters, no baseline condition (e.g., concatenation without the outpainting operation, shuffled prompt order, or prompts not derived from the score), and no quantitative indicator such as CLAP-based text-audio alignment or a structural similarity measure between segments. Because these claims are the paper's headline, they need stronger support; a small perceptual experiment or a systematic ablation with reported audio examples would be sufficient to move the paper from anecdotal to evidential.
- [Section II-B, Eq. (1)] The 'outpainting' operation in Eq. (1) feeds the pretrained MusicLDM diffusion model f an input whose first half is a clean/denoised latent z_k and whose second half is Gaussian noise. MusicLDM's f was trained to denoise standard Gaussian noise, so this mixed input is out-of-distribution and may produce artifacts unrelated to a smooth continuation; the paper offers no validation, ablation, or quantitative check of this mechanism. The cited outpainting methods [18], [19] keep known context throughout the denoising trajectory rather than only in the initial latent, so the analogy is not a standard extension. Please provide evidence that the operation behaves as intended, for example spectrograms with and without Eq. (1), or a continuity metric computed across segment boundaries.
- [Appendix A] The reconstruction step is under-specified. The text says each new segment begins with the second half of the previous latent, but it also says the final waveform is obtained by concatenating the denoised latent representations. If both statements hold, each 10-second segment contains a copy of the previous segment's second half, and simple concatenation reproduces the overlapped content twice; the paper does not state whether the overlap is discarded, averaged, cross-faded, or otherwise masked. This ambiguity affects the reproducibility of the central 'seamless' claim. Please specify the exact stitching rule and reconcile the mask m in R^(8x128x16) with the T//2 split in Eq. (1), given that the latent has T=256 time steps.
minor comments (6)
- [Abstract] The abstract contains a typo: 'demostrate' should be 'demonstrate'.
- [Section II-B] The notation in Eq. (1) defines epsilon-prime, but the text continues to call it a 'noise input'; clarify that the input is no longer pure Gaussian noise and explain how the diffusion sampling trajectory is initialized under this mixed input.
- [Appendix D] Page 2 lists the same prompt four times; if this is intentional, explain why identical prompts yield four distinct segments, or correct the apparent typo.
- [Section II-B] The term 'outpainting' is used for an operation that is closer to latent continuation; since [18] and [19] use outpainting with context maintained throughout sampling, the terminology may mislead readers. Consider a more precise term such as 'latent overlap continuation'.
- [Appendix A] The paper does not report how many ChatGPT runs were used, whether the generated prompts were filtered or selected, or how the keyword prefixes were chosen; please include these details for reproducibility.
- [Fig. 1] The pipeline figure does not show how the overlap is resolved after decoding; adding a clear diagram of the stitching and overlap-handling step would make the method easier to follow.
Circularity Check
No significant circularity: the pipeline applies pretrained models to caption-derived prompts, and Eq. (1) defines a stitching heuristic rather than fitting a target.
full rationale
The claimed derivation chain is ChatGPT captioning of Treatise pages, CLAP text embedding, MusicLDM latent diffusion, and concatenative overlap via Eq. (1). No step fits a parameter to a quantity that is later reported as a prediction. Eq. (1) sets epsilon' = concat(z_k[:, T//2:, :], epsilon[:, :T//2, :]); this is an input-construction rule for the next diffusion call, not an equation whose output is forced by a fitted input. The paper's 'seamless and cohesive composition' claim depends on an unvalidated empirical premise that MusicLDM behaves gracefully on half-clean latent inputs, but an unsupported premise is a soundness/evaluation limitation, not circularity. The authors' qualitative judgments are self-referential in that the same authors wrote the prompts and assessed score-following, but that is an evaluation-validity weakness rather than an equivalence-by-construction. Self-citations to MusicLDM [10] and CLAP [24] are to public pretrained checkpoints and are used as components, not as unverified authority justifying the central claim; under the stated rules these are real independent evidence and do not raise the score. No fitted-input-called-prediction, uniqueness import, or ansatz-smuggling-via-citation is present.
Assumptions & free parameters
free parameters (2)
- Latent overlap fraction (T//2) =
50% (128 of 256 latent time steps)
- User keyword prefixes =
various strings, e.g. 'sine wave is playing', 'string orchestra is playing', 'Experimental music is playing'
assumptions (3)
- domain assumption ChatGPT 4o's text prompts accurately capture the musically relevant visual structure of Treatise pages.
- ad hoc to paper The latent diffusion model f accepts a half-latent, half-noise initial input and remains on the data manifold.
- domain assumption CLAP's text-audio embedding space aligns the generated text prompts with the acoustic qualities MusicLDM can synthesize.
Cite this review
Pith. "Pith review of Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatise." pith.science (2026). https://pith.science/paper/SIOV6VYS
@misc{pith2026241208944,
author = {Pith},
title = {Pith review of: Interpreting Graphic Notation with MusicLDM: An AI Improvisation of Cornelius Cardew's Treatise},
year = {2026},
howpublished = {\url{https://pith.science/paper/SIOV6VYS}},
note = {Machine review of arXiv:2412.08944}
}
read the original abstract
This work presents a novel method for composing and improvising music inspired by Cornelius Cardew's Treatise, using AI to bridge graphic notation and musical expression. By leveraging OpenAI's ChatGPT to interpret the abstract visual elements of Treatise, we convert these graphical images into descriptive textual prompts. These prompts are then input into MusicLDM, a pre-trained latent diffusion model designed for music generation. We introduce a technique called "outpainting," which overlaps sections of AI-generated music to create a seamless and cohesive composition. We demostrate a new perspective on performing and interpreting graphic scores, showing how AI can transform visual stimuli into sound and expand the creative possibilities in contemporary/experimental music composition. Musical pieces are available at https://bit.ly/TreatiseAI
Figures
Reference graph
Works this paper leans on
-
[18]
MultiDiffusion: Fus- ing diffusion paths for controlled image generation,
O. Bar-Tal, L. Yariv, Y . Lipman, and T. Dekel, “MultiDiffusion: Fus- ing diffusion paths for controlled image generation,” in International Conference on Machine Learning (ICML) , 2023
work page 2023
-
[19]
Controllable music production with diffusion models and guidance gradients,
M. Levy, B. D. Giorgi, F. Weers, A. Katharopoulos, and T. Nickson, “Controllable music production with diffusion models and guidance gradients,” ArXiv, vol. abs/2311.00613, 2023
arXiv 2023
-
[1]
WaveNet: A Generative Model for Raw Audio,
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A Generative Model for Raw Audio,” in Proc. ISCA, 2016, p. 125
work page 2016
-
[2]
Samplernn: An unconditional end-to-end neural audio generation model,
S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. C. Courville, and Y . Bengio, “Samplernn: An unconditional end-to-end neural audio generation model,” in ICLR, 2017
work page 2017
-
[3]
Adversarial audio synthe- sis,
C. Donahue, J. McAuley, and M. Puckette, “Adversarial audio synthe- sis,” in ICLR, 2019
work page 2019
-
[4]
Jukebox: A generative model for music,
P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” arXiv:2005.00341, 2020
arXiv 2005
-
[5]
Musiclm: Generating music from text,
A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv:2301.11325, 2023
arXiv 2023
-
[6]
Simple and controllable music generation,
J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music generation,” in NeurIPS, 2023
work page 2023
Show all 24 references
-
[7]
Mustango: Toward controllable text-to-music generation,
J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria, “Mustango: Toward controllable text-to-music generation,” in NAACL, 2024
2024
-
[8]
Mo ˆusai: Text-to- music generation with long-context latent diffusion,
F. Schneider, O. Kamal, Z. Jin, and B. Sch ¨olkopf, “Mo ˆusai: Text-to- music generation with long-context latent diffusion,” arXiv:2301.11757, 2023
2023 arXiv
-
[9]
Diffwave: A versatile diffusion model for audio synthesis,
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in ICLR, 2021
2021
-
[10]
Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies,
K. Chen, Y . Wu, H. Liu, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov, “Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies,” in ICASSP. IEEE, 2024, pp. 1206–1210
2024
-
[11]
Enabling factorized piano music modeling and generation with the maestro dataset,
C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the maestro dataset,” arXiv preprint arXiv:1810.12247, 2018
-
[12]
Multi-instrument music synthesis with spectrogram diffusion,
C. Hawthorne, I. Simon, A. Roberts, N. Zeghidour, J. Gardner, E. Manilow, and J. Engel, “Multi-instrument music synthesis with spectrogram diffusion,” arXiv preprint arXiv:2206.05408 , 2022
2022 arXiv
-
[13]
Diffuseroll: Multi-track multi-category music generation based on diffusion model,
H. Wang, “Diffuseroll: Multi-track multi-category music generation based on diffusion model,” arXiv preprint arXiv:2303.07794 , 2023
2023 arXiv
-
[14]
Draw and listen! a sketch-based system for music inpainting,
C. Benetatos and Z. Duan, “Draw and listen! a sketch-based system for music inpainting,” Transactions of the International Society for Music Information Retrieval , Nov 2022
2022
-
[15]
Music controlnet: Multiple time-varying controls for music generation,
S.-L. Wu, C. Donahue, S. Watanabe, and N. J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” arXiv:2311.07069, 2023
2023 arXiv
-
[16]
Ditto: Diffusion inference-time t-optimization for music generation,
Z. Novack, J. McAuley, T. Berg-Kirkpatrick, and N. J. Bryan, “Ditto: Diffusion inference-time t-optimization for music generation,” arXiv:2401.12179, 2024
2024 arXiv
-
[17]
Polyffusion: A diffusion model for polyphonic score generation with internal and external controls,
L. Min, J. Jiang, G. Xia, and J. Zhao, “Polyffusion: A diffusion model for polyphonic score generation with internal and external controls,” in Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 202...
2023
-
[20]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017
2017
-
[21]
Audioldm 2: Learning holistic audio generation with self-supervised pretraining,
H. Liu, Q. Tian, Y . Yuan, X. Liu, X. Mei, Q. Kong, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” arXiv:2308.05734, 2023
2023 arXiv
-
[22]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, vol. 33, 2020, pp. 6840–6851
2020
-
[23]
Deep unsupervised learning using nonequilibrium thermodynamics,
J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML, vol. 37, 2015, pp. 2256–2265
2015
-
[24]
Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,
Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc. ICASSP, 2023, pp. 1–5. APPENDIX A COMPOSITIONAL SETUP A. Score Interpretation with Cha...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.