REVIEW 4 major objections 5 minor 20 references
Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A new model for classical piano generation argues that composing and performing should be two separate but jointly trained branches, with a compound token that captures per-note expressive timing.
desk verdict A useful engineering contribution to symbolic piano generation whose headline claim of 'superior musical quality' outruns the statistical evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Expressive Compound Word (ECP) representation, a compound token that combines four score sub-tokens and four performance sub-tokens at every time step, so that metrical position and note-level expressive timing are represented together. The model architecture is a two-branch Seq2Seq framework: a VQ-VAE-based Composer branch that generates the score-related sub-tokens, and a vanilla VAE-based Pianist branch that produces the expressive sub-tokens, with both branches using a multiscale encoder that adds beat-level self-attention to standard self-attention and an orthogonal Transformer decoder that first decodes along the temporal axis and then along the sub-token axis. This decomposition is what allows the model to represent what is played and how it is played separately while keeping the two coupled during joint training.
What would settle it
Take a real performance, cut it into bars, shuffle the bars, and re-quantize the result so its UPC, PR, APS, DSTD, DS, IOI, and AVI values stay close to real-music values; if the seven objective metrics rate this shuffled piece as comparable to XMVAE's output while human listeners rate it as incoherent, then closeness to corpus statistics is not sufficient evidence for the paper's quality claim.
Extended reading notes
Core claim
The central claim is that score content and expressive performance should be generated by two different latent-variable mechanisms that are trained together, rather than by a single architecture or by independent sequential stages. The Composer branch uses a VQ-VAE to learn a discrete codebook of note-level musical ideas, while the Pianist branch uses a vanilla VAE to sample a global performance style from continuous expressive parameters and then decodes expressive tokens while conditioned on the composer's code sequence. Because the two branches share the same quantized latent sequence at training time, the performer is guided by the actual composed content instead of by a separately sampled score. The paper takes its empirical support to be the closeness of XMVAE output to real-music statistics on pitch, rhythm, and performance metrics, plus a 26-subject listening test in which XMVAE leads all benchmark models on coherency, richness, and overall quality while still trailing real human performances.
Load-bearing premise
The paper's main evidence for better musical quality is that generated samples' aggregate statistics sit closer to real performances' aggregate statistics, which assumes that matching these global numbers means the music actually sounds coherent and musical.
Editorial extensions
If this is right
- A representation that records beat position inside each token can support note-level expressive timing in symbolic classical music, rather than only beat-level tempo tokens.
- Pretraining the composer branch on larger score-only datasets is a workable way to improve expressive generation when aligned score-performance data are scarce.
- Jointly training content and style branches can avoid the sampling errors the paper attributes to two-stage generation, where a performer is conditioned on a separately sampled score.
- The same architecture may transfer to performance rendering by fixing the composer latent sequence of a given score and letting the performance decoder interpret it, as the paper flags for future work.
Reading between the lines
- The paper leaves implicit that its evaluation would be strengthened by a coherence-sensitive measure: because all seven objective metrics are corpus-level aggregates, a model that matches them could still fail on local musical structure, and the small listening test does not yet rule that out.
- A natural testable extension is to vary the amount of rubato in the test pieces and check whether removing the real-valued expressive input $X^{pv}$ hurts most on pieces with heavy tempo fluctuation, which would isolate what that input contributes.
- The ECP token and two-branch design are piano-specific in their current performance tokens, but the same separation of score and expression could plausibly be carried to other instruments or small ensembles by redefining the four performance sub-tokens.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a symbolic classical piano performance generation system from scratch. It introduces the Expressive Compound Word (ECP) tokenization, which combines four score-related sub-tokens and four performance-related sub-tokens per time step, and the Expressive Music Variational AutoEncoder (XMVAE), a two-branch model with a VQ-VAE-based Composer branch for score content and a vanilla VAE-based Pianist branch for expressive details. The two branches are trained jointly, with a multiscale encoder and an orthogonal Transformer decoder. The authors compare XMVAE against Music Transformer, Transformer-GAN, ECPT, and NMT on seven objective aggregate statistics and a 26-participant listening test, and they report ablation studies showing the benefit of pretraining the Composer branch on additional score data. The central claim is that XMVAE produces classical piano performances of superior musical quality compared to state-of-the-art models.
Significance. If the central claim were fully established, this would be a useful contribution to symbolic music generation. The two-branch composer/pianist decomposition is a natural inductive bias, the ECP representation is compact and explicitly designed for note-level expressive timing, and the joint training of the two branches is a plausible way to avoid the sampling error of independently trained sequential models. The ablation studies are valuable because they isolate the multiscale encoder, the sub-token decoder, and the real-valued performance inputs, and the pretraining result is interesting. The paper also gives enough architectural detail to re-implement the model. However, the main comparative claim is not currently supported at the strength asserted: the objective metrics are point estimates without uncertainty, the subjective test has overlapping confidence intervals and no significance tests, and the listening test uses a pretrained variant while the main objective table uses a non-pretrained variant. These issues are load-bearing for the paper's headline conclusion.
major comments (4)
- [Section IV.B, Table I] The objective evaluation reports only point estimates for each of the seven metrics, computed over 1,000 generated samples per model, with no error bars, confidence intervals, or significance tests. Consequently, the claim that XMVAE is best on six of seven metrics is not distinguished from sampling noise. Moreover, the paper states that 'Models with metric scores closer to those of the training dataset are considered to perform better,' but this assumption is not validated against perceptual judgments; a model could match aggregate statistics while producing locally incoherent music. The table also shows that on APS, NMT is closer to Real Music than XMVAE, so the advantage is not uniform across metrics. I recommend reporting variances or bootstrap confidence intervals for all metrics, running significance tests on the differences, and, ideally, validating the chosen metrics against human ratings.
- [Section IV.C, Table III] The subjective listening test uses only 26 participants and reports 95% confidence intervals that overlap substantially across models. For example, on the Overall criterion, XMVAE(pretrained) scores 3.55 ± 0.43 while NMT scores 3.14 ± 0.58, and on Coherency the intervals are 3.38 ± 0.43 versus 2.98 ± 0.66. With overlapping confidence intervals and no significance testing, the statement that XMVAE 'outperforms other benchmark models across all criteria' is not statistically supported. I ask for pairwise significance tests (with multiple-comparison correction) and an explicit report of effect sizes or at least the test statistics.
- [Section IV.B versus Section IV.C] The main objective comparison in Table I uses the non-pretrained XMVAE, while the listening test in Table III uses the pretrained XMVAE. This is an inconsistency: the subjective evidence for 'superior musical quality' may reflect the pretraining gain rather than the proposed architecture itself, and the objective and subjective results are not directly comparable. The paper should either include both variants in both evaluations or clearly state which configuration is being compared in each table and justify why the comparison is still meaningful.
- [Section IV.C] The listening test conditions all generated samples on a prime consisting of the first beat of the corresponding score from Real Music. This changes the task from 'performance generation from scratch,' which is the stated focus of Sections I and V, to a continuation or conditioned-generation task. This is not necessarily a flaw, but it should be acknowledged, and an additional unprimed evaluation would strengthen the link between the test and the paper's main claim.
minor comments (5)
- [Section II] The sentence 'Recent study like Cadenza [9]' appears to cite reference [9], which is the PerTok paper by Lenz and Mani; if Cadenza is a different system, the citation is incorrect or missing.
- [Section IV.B] The text says metrics should be close to 'those of the training dataset,' but Real Music values are computed from the testing dataset. Please clarify which dataset is used for the reference row.
- [Section III.B.3] Equation (6) sums the Composer and Pianist losses without a weighting parameter or a discussion of their relative scales. A short comment on why a plain sum is appropriate would help.
- [Section III.B.3] The prior distribution is written as $p_{ψ}(z_s) = Q_i p_{ψ}(z_{s,i} | z_{s,<i})$; the symbol 'Q' should be the product symbol '∏'.
- [Table III caption] The caption reads 'Subjective Listening Rest Results'; 'Rest' should be 'Test.'
Circularity Check
No significant circularity: XMVAE is trained on reconstruction losses and evaluated on held-out statistics and human ratings; no fitted parameter is renamed as a prediction.
full rationale
The paper's central claim is an empirical comparison: XMVAE is trained with the VQ-VAE and vanilla VAE losses in Eqs. (4)-(6), plus an autoregressive prior over the codebook, and then evaluated on 1,000 generated samples per model against seven aggregate statistics (Table I) and a listening test (Table III). None of the seven metrics (UPC, PR, APS, DSTD, DS, IOI, AVI) is a training objective or a fitted parameter; they are computed from held-out real recordings and from generated outputs after training. The 'closer to Real Music is better' convention comes from prior work [6], [7] and is a measurement choice, not a construction that forces XMVAE's outputs to match. The listening test uses independent human ratings with reported confidence intervals. The two self-citations ([2], [12]) are background and duration-quantization details respectively; neither is invoked to justify the model design or to forbid alternatives. No uniqueness theorem from prior work is imported, no ansatz is justified solely by a same-author citation, and no known result is merely renamed: ECP is a concrete tokenization built on CP, and the two-branch VQ-VAE/VAE architecture is trained jointly rather than derived from the evaluation metrics. Therefore the claimed superiority over NMT, MT, Transformer-GAN, and ECPT, whatever its statistical strength, is not circular.
Assumptions & free parameters
free parameters (5)
- Beat resolution (ticks per beat) =
24
- Quantization bins for Beat Period, Timing, Articulation, Velocity =
161, 41, 81, 32
- Codebook size K for VQ-VAE =
512
- Latent dimension and model dimensions =
hidden 256, latent 512, 6 layers, 8 heads, FFN 1024
- Loss weights α and β =
not reported
assumptions (3)
- domain assumption The filtered ATEPP subset provides accurate score-performance alignment.
- domain assumption A 512-entry codebook can represent the score content sufficiently for the Pianist branch.
- domain assumption Aggregate statistics close to the training set indicate musical quality.
Cite this review
Pith. "Pith review of Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder." pith.science (2026). https://pith.science/paper/LE7IIARI
@misc{pith2026250701582,
author = {Pith},
title = {Pith review of: Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder},
year = {2026},
howpublished = {\url{https://pith.science/paper/LE7IIARI}},
note = {Machine review of arXiv:2507.01582}
}
read the original abstract
The creativity of classical music arises not only from composers who craft the musical sheets but also from performers who interpret the static notations with expressive nuances. This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist in the creative process. We introduce the Expressive Compound Word (ECP) representation, which effectively captures both the metrical structure and expressive nuances of classical performances. Building on this, we propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, representing the Composer, and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist. These branches are jointly trained with similar Seq2Seq architectures, leveraging a multiscale encoder to capture beat-level contextual information and an orthogonal Transformer decoder for efficient compound tokens decoding. Both objective and subjective evaluations demonstrate that XMVAE generates classical performances with superior musical quality compared to state-of-the-art models. Furthermore, pretraining the Composer branch on extra musical score datasets contribute to a significant performance gain.
Figures
Reference graph
Works this paper leans on
-
[1]
Computational models of expressive music performance: A compre- hensive and critical review,
C. E. Cancino-Chac ´on, M. Grachten, W. Goebl, and G. Widmer, “Computational models of expressive music performance: A compre- hensive and critical review,” Frontiers in Digital Humanities , vol. 5, p. 25, 2018
work page 2018
-
[2]
S. Ji, X. Yang, and J. Luo, “A survey on deep learning for symbolic music generation: Representations, algorithms, evaluations, and chal- lenges,” ACM Computing Surveys , vol. 56, no. 1, pp. 1–39, 2023
work page 2023
-
[3]
Virtuosonet: A hierarchical rnn-based system for modeling expressive piano perfor- mance
D. Jeong, T. Kwon, Y . Kim, K. Lee, and J. Nam, “Virtuosonet: A hierarchical rnn-based system for modeling expressive piano perfor- mance.” in ISMIR, 2019, pp. 908–915
work page 2019
-
[4]
Dexter: Learning and controlling performance ex- pression with diffusion models,
H. Zhang, S. Chowdhury, C. E. Cancino-Chac ´on, J. Liang, S. Dixon, and G. Widmer, “Dexter: Learning and controlling performance ex- pression with diffusion models,” Applied Sciences, vol. 14, no. 15, p. 6543, 2024
work page 2024
-
[5]
This time with feeling: Learning expressive musical performance,
S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan, “This time with feeling: Learning expressive musical performance,” Neural Computing and Applications , vol. 32, pp. 955–967, 2020. Fig. 3. Visualized pianoroll of real music and musical samples generated by different models
work page 2020
-
[6]
Symbolic music generation with transformer-gans,
A. Muhamed, L. Li, X. Shi, S. Yaddanapudi, W. Chi, D. Jackson, R. Suresh, Z. C. Lipton, and A. J. Smola, “Symbolic music generation with transformer-gans,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 1, 2021, pp. 408–417
work page 2021
-
[7]
Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,
Y .-S. Huang and Y .-H. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in Proceedings of the 28th ACM international conference on multimedia , 2020, pp. 1180–1188
2020
-
[8]
Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,
W.-Y . Hsiao, J.-Y . Liu, Y .-C. Yeh, and Y .-H. Yang, “Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 1, 2021, pp. 178–186
2021
Show all 20 references
-
[9]
Pertok: Expressive encoding and modeling of symbolic musical ideas and variations,
J. Lenz and A. Mani, “Pertok: Expressive encoding and modeling of symbolic musical ideas and variations,” in ISMIR, 2024, pp. 1–8
2024
-
[10]
Music transformer: Generating music with long-term structure,
C. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music transformer: Generating music with long-term structure,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, ...
2019
-
[11]
Disentangling the horowitz factor: Learning content and style from expressive piano performance,
H. Zhang and S. Dixon, “Disentangling the horowitz factor: Learning content and style from expressive piano performance,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5
2023
-
[12]
Bandcontrolnet: Parallel transformers-based steerable popular music generation with fine- grained spatiotemporal features,
J. Luo, X. Yang, and D. Herremans, “Bandcontrolnet: Parallel transformers-based steerable popular music generation with fine- grained spatiotemporal features,” arXiv preprint arXiv:2407.10462 , 2024
2024
-
[13]
Partitura: A Python Package for Symbolic Music Processing,
C. E. Cancino-Chac ´on, S. D. Peter, E. Karystinaios, F. Foscarin, M. Grachten, and G. Widmer, “Partitura: A Python Package for Symbolic Music Processing,” in Proceedings of the Music Encoding Conference (MEC2022), Halifax, Canada, 2022
2022
-
[14]
Neural discrete representation learning,
A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017
2017
-
[15]
Auto-encoding variational bayes,
D. P. Kingma, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013
2013 arXiv
-
[16]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[17]
Figaro: Controllable music generation using learned and expert features,
D. von R ¨utte, L. Biggio, Y . Kilcher, and T. Hofmann, “Figaro: Controllable music generation using learned and expert features,” in The Eleventh International Conference on Learning Representations , 2023, pp. 1–18
2023
-
[18]
Nested music transformer: Sequentially decoding compound tokens in symbolic music and audio generation,
J. Ryu, H.-W. Dong, J. Jung, and D. Jeong, “Nested music transformer: Sequentially decoding compound tokens in symbolic music and audio generation,” in ISMIR, 2024, pp. 1–8
2024
-
[19]
Representation learning with contrastive predictive coding,
A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018
2018 arXiv
-
[20]
Atepp: A dataset of automatically transcribed expressive piano performance,
H. Zhang, J. Tang, S. R. Rafee, S. Dixon, G. Fazekas, and G. A. Wiggins, “Atepp: A dataset of automatically transcribed expressive piano performance,” in Ismir 2022 Hybrid Conference, 2022, pp. 446– 453
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.