REVIEW 4 major objections 5 minor 32 references
Detecting Music Performance Errors with Transformers
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Polytune is an end-to-end transformer that detects music errors by comparing score and performance audio without explicit alignment.
desk verdict Solid engineering, but the headline F1 is measured entirely on synthetic test sets from the same pipeline that made the training data; the 40-point gain is real on that distribution, not yet on real performances. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a dual-encoder transformer. Two Audio Spectrogram Transformer (AST) encoders process the two input spectrograms separately—one for score audio, one for performance audio—and their outputs are concatenated and fed through a joint encoder, with a T5-style decoder generating tokens via greedy autoregressive sampling. The token vocabulary is MIDI-like (time, on/off, pitch) with the addition of a Label token for each note, so each decoded event is explicitly Correct, Missed, or Extra. Training data come from Algorithm 1, which injects errors into MIDI files with a Poisson-selected note rate and truncated-normal offsets for pitch and time, then synthesizes audio with MIDI-DDSP; a weighted cross-entropy loss with weight 10 on error tokens handles the class imbalance. This combination lets the model learn alignment implicitly rather than through a separate DTW stage.
What would settle it
Gather recordings of real beginner students playing from the same scores, have expert musicians label every note as correct, missed, or extra, and run the trained Polytune model on those recordings without fine-tuning. If its missed-note and extra-note F1 scores fall to roughly 30% or below, or below the DTW baseline, the claim that the method generalizes beyond synthetic data would be refuted.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that explicit alignment is not a necessary component of music error detection. Polytune learns to compare the score audio and performance audio in a joint latent space, and its output vocabulary is the familiar MIDI-like token sequence augmented with three label tokens: Correct, Missed, and Extra. Averaged over 14 instruments, the model reports F1 scores of 95.0% for correct notes, 49.2% for missed notes, and 48.0% for extra notes, with an overall Error Detection F1 of 64.1% that beats the reimplemented DTW-based baselines by 40 percentage points. The same model, without instrument-specific tokens, works on all tested instruments, which the authors present as evidence that a single end-to-end transformer can serve as a general performance-error annotator.
Load-bearing premise
The load-bearing assumption is that the synthetic pipeline—errors injected into MIDI and rendered with one neural synthesizer per instrument—produces performances that are representative enough of real student mistakes; if real errors differ in timbre, timing, or type, the reported F1 scores will not transfer.
Editorial extensions
If this is right
- An end-to-end transformer can annotate correct, missed, and extra notes without an explicit alignment stage, so alignment-induced misclassifications disappear from the pipeline.
- The synthetic error injection algorithm turns existing MIDI transcription datasets into large error-detection datasets, removing the data scarcity that previously limited the task to a handful of tracks.
- Because the output vocabulary is instrument-agnostic, one trained model can detect errors across 14 instruments rather than requiring a separate system per instrument.
- Since the inputs are raw spectrograms, the same architecture can be extended to timing or dynamics errors without changing the input representation.
- The reported 40-point average F1 improvement over reimplemented DTW baselines indicates that learned implicit alignment can outperform explicit time warping on this task.
Reading between the lines
- Editorial inference: the dual-encoder 'reference audio versus query audio' design is generic; the same approach could be adapted to singing assessment, pronunciation tutoring, or any task where one recording is compared to a reference recording.
- Editorial inference: the large gap between correct-note F1 (95.0%) and missed-note F1 (49.2%) suggests the model is much better at verifying played notes than at detecting omissions; a score-aware auxiliary objective might close that gap.
- Editorial inference: because the error-injection distribution defines what counts as an error, real student errors with different statistics (such as rhythmic anticipation or repeated notes) could require retraining with additional error types rather than transferring directly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Polytune, an end-to-end transformer model for detecting performance errors in music. The model takes two audio spectrogram streams as input—one synthesized from a reference score and one from the student performance—and outputs MIDI-like tokens annotated as Correct, Missed, or Extra. The authors introduce Algorithm 1, which injects synthetic errors (missed notes, pitch changes, timing shifts, extra notes) into existing MIDI datasets, and they render the augmented files with MIDI-DDSP to create MAESTRO-E and CocoChorales-E. They compare Polytune against a reimplemented MT3-plus-DTW baseline on 4,401 synthetic test tracks, reporting average Error Detection F1 of 64.1%, roughly 40 percentage points above the baseline. The paper claims that Polytune achieves state-of-the-art music error detection without an explicit alignment stage.
Significance. If the claims hold, the work is significant: it demonstrates a learned, implicit alignment mechanism for score-informed error detection, scales training data through synthetic error injection, and extends error detection to 14 instruments. The public release of code and datasets is a genuine strength, as is the detailed description of the architecture and data generation pipeline. However, the significance is conditional on whether the synthetic pipeline transfers to real performances. The evaluation is entirely on renderings produced by the same MIDI-DDSP pipeline used for training, and the baseline is a self-reimplementation rather than original code or published results. These two issues mean that the headline numbers currently establish performance on a synthetic distribution, not on the real student-practice setting the paper motivates. The central idea is promising and the main missing piece—real-data validation or a clear reframing of the claims—is addressable.
major comments (4)
- [§3.3 and §5] The evaluation is restricted to synthetic test sets generated by the same pipeline (Algorithm 1 and MIDI-DDSP synthesis) used to create the training data, and Section 5 concedes that using only one synthesizer per instrument may restrict generalization. Consequently, the claims of 'state-of-the-art music error detection' and the 40-percentage-point improvement are, as presented, claims about MIDI-DDSP renderings rather than real performances. The paper should either validate on real recordings (e.g., the small Benetos et al. dataset or newly recorded student performances with annotations) or substantially narrow the stated claims to synthetic benchmarks. This is the load-bearing point for the paper's application setting.
- [§3.4 and Tables 3–4] The baseline is a new reimplementation—MT3 with DTW—not the original code or results from Benetos et al. or Wang et al., and no evidence is provided that the reimplementation reproduces or approximates the published methods. Because the headline '40 percentage point improvement' is computed relative to this self-reimplementation, the comparison is weakened. Please report the baseline's hyperparameters, MT3 initialization, DTW settings, and, if possible, validation against the original published approaches on a shared dataset.
- [§4.1 and §4.2] No error bars, confidence intervals, or multi-seed runs are reported. The F1 differences, especially for Missed and Extra classes, could be sensitive to random dataset generation, model initialization, or decoding randomness. Please report variance across at least a few training runs or bootstrap confidence intervals on the test set, and indicate how many tracks or notes each F1 value is based on.
- [§3.3, Algorithm 1 and Output Data] Algorithm 1 injects pitch-change (wrong note) and timing-shift errors, but the output vocabulary in Table 2 contains only Correct, Missed, and Extra labels. The paper should explain how pitch-change and timing-shift errors are converted into these label tokens and how they are counted in the evaluation. Without this operationalization, the 'wrong note' behavior described in Figure 1 is not directly measurable from the reported metrics.
minor comments (5)
- [Abstract] The phrase 'alignment targets.;' contains a stray semicolon and should be corrected.
- [§3.3] The text reads 'two truncated normal distribution distributions' and should be 'two truncated normal distributions.'
- [Table 1] The row label 'Y ousician' contains an extra space; it should read 'Yousician.'
- [Table 1 and §3.4] Table 1 lists a 'Combined' baseline, but Section 3.4 does not define how this baseline combines Benetos et al. and Wang et al. Please add a definition or remove the entry.
- [Figures 3 and 5] The captions refer to 'Music note 1' and 'Music note 2' and to specific pitches such as 'A' and 'C#', but the figures do not clearly mark these notes; please add explicit labels to the figures.
Circularity Check
No circularity: Polytune's synthetic-data evaluation is a genuine held-out generalization measure, not a fitted input renamed as a prediction.
full rationale
Polytune's error detection is trained and tested on CocoChorales-E and MAESTRO-E, with both train and test examples produced by Algorithm 1. This shared generative pipeline raises external-validity concerns (the paper itself notes the use of only one synthesizer per instrument in Sec. 5), but it is not a circular derivation: the model never receives the error-injection parameters at inference, the test set is disjoint from training, and the reported F1 is a genuine generalization measure on that distribution. The baseline is an upgraded re-implementation of prior work, not an input to Polytune. No load-bearing self-citation, imported uniqueness, renamed known result, or fitted-parameter-relabeled-as-prediction appears. The central comparison is self-contained against the synthetic benchmarks, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Error rate lambda =
sampled from U(0.1, 0.4)
- Pitch offset distribution P =
truncated normal, mean 0, SD 1
- Timing offset distribution Q =
truncated normal, mean 0, SD 0.02
- Segment length =
2.145 seconds
- Error class weight alpha =
10
assumptions (3)
- domain assumption MIDI-DDSP synthesized audio adequately represents real instrument performances for error detection.
- domain assumption Algorithm 1's Poisson selection and truncated normal offsets produce realistic performance errors.
- domain assumption The reimplemented MT3 plus DTW baseline fairly represents prior score-informed error detection systems.
Cite this review
Pith. "Pith review of Detecting Music Performance Errors with Transformers." pith.science (2026). https://pith.science/paper/U7NKNJFR
@misc{pith2026250102030,
author = {Pith},
title = {Pith review of: Detecting Music Performance Errors with Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/U7NKNJFR}},
note = {Machine review of arXiv:2501.02030}
}
read the original abstract
Beginner musicians often struggle to identify specific errors in their performances, such as playing incorrect notes or rhythms. There are two limitations in existing tools for music error detection: (1) Existing approaches rely on automatic alignment; therefore, they are prone to errors caused by small deviations between alignment targets.; (2) There is a lack of sufficient data to train music error detection models, resulting in over-reliance on heuristics. To address (1), we propose a novel transformer model, Polytune, that takes audio inputs and outputs annotated music scores. This model can be trained end-to-end to implicitly align and compare performance audio with music scores through latent space representations. To address (2), we present a novel data generation technique capable of creating large-scale synthetic music error datasets. Our approach achieves a 64.1% average Error Detection F1 score, improving upon prior work by 40 percentage points across 14 instruments. Additionally, compared with existing transcription methods repurposed for music error detection, our model can handle multiple instruments. Our source code and datasets are available at https://github.com/ben2002chou/Polytune.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Akbari, H.; Chuang, W.-H.; Yuan, L.; Chang, S.-F.; Gong, B.; Qian, R.; and Cui, Y. 2021. VATT : Transformers for Multimodal Self - Supervised Learning from Raw Video , Audio and Text . In 35th Conference on Neural Information Processing Systems
work page 2021
-
[4]
Apaydınlı, K. 2019. Intelligent Tutoring Systems in Music Education . In Proceedings of The 2nd International Conference on Future of Teaching and Education . GLOBALKS. ISBN 978-609-485-043-1
work page 2019
-
[5]
Benetos, E.; Klapuri, A.; and Dixon, S. 2012. Score-informed transcription for automatic piano tutoring. Proceedings of the European Signal Processing Conference (EUSIPCO), pp. 2153--2157
work page 2012
-
[6]
Britannica, T. E. o. E. 2007. Homophony
work page 2007
-
[7]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale . In Proceedings of the International Conference on Learning Representations
work page 2021
-
[8]
Ewert, S.; Wang, S.; and Sandler, M. 2016. Score-informed Identification of Missing and Extra Notes in Piano Recordings . Proceedings of the International Conference on Music Information Retrieval (ISMIR)
work page 2016
Show all 32 references
-
[9]
Fradet, N.; Briot, J.-P.; and Chhel, F. 2021. MidiTok : A Python package for MIDI file tokenization. Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference
2021
-
[10]
Gardner, J.; Simon, I.; Manilow, E.; Hawthorne, C.; and Engel, J. 2022. MT3 : Multi - Task Multitrack Music Transcription . International Conference on Learning Representations
2022
-
[11]
Gong, Y.; Chung, Y.-A.; and Glass, J. 2021. AST : Audio Spectrogram Transformer . In Interspeech 2021, 571--575. ISCA
2021
-
[12]
H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J
Gong, Y.; Rouditchenko, A.; Liu, A. H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J. 2023. Contrastive Audio - Visual Masked Autoencoder . In Proceedings of the International Conference on Learning Representations ( ICLR )
2023
-
[13]
Hawthorne, C.; Simon, I.; Swavely, R.; Manilow, E.; and Engel, J. 2021. Sequence-to-sequence piano transcription with transformers. In Proceedings of the 22nd ISMIR Conference
2021
-
[14]
A.; Dieleman, S.; Elsen, E.; Engel, J.; and Eck, D
Hawthorne, C.; Stasyuk, A.; Roberts, A.; Simon, I.; Huang, C.-Z. A.; Dieleman, S.; Elsen, E.; Engel, J.; and Eck, D. 2018. Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset . In Proceedings of the International Conference on Learning Representations
2018
-
[15]
Hsiao, W.-Y.; Liu, J.-Y.; Yeh, Y.-C.; and Yang, Y.-H. 2021. Compound Word Transformer : Learning to Compose Full - Song Music over Dynamic Directed Hypergraphs . Proceedings of the AAAI Conference on Artificial Intelligence, 35(1): 178--186
2021
-
[16]
Huang, P.-Y.; Xu, H.; Li, J.; Baevski, A.; Auli, M.; Galuba, W.; Metze, F.; and Feichtenhofer, C. 2022. Masked Autoencoders that Listen . NeurIPS
2022
-
[17]
Huang, Y.-S.; and Yang, Y.-H. 2020. Pop Music Transformer : Beat -based Modeling and Generation of Expressive Pop Piano Compositions . In Proceedings of the 28th ACM International Conference on Multimedia , 1180--1188. Seattle WA USA: ACM. ISBN 978-1-4503-7988-5
2020
-
[18]
JoyTunes. 2024. Simply Piano
2024
-
[19]
B.; McCormick, P.; Shepherd, J
Morrison, R. B.; McCormick, P.; Shepherd, J. L.; and Cirillo, P. 2022. National Arts Education Status Report Summary 2019. Technical report, Arts Education Data Project, Quadrant Research, State Education Agency Directors of Arts Education
2022
-
[20]
Nart, S. 2016. Music Software in the Technology Integrated Music Education . The Turkish Online Journal of Educational Technology, 15(2)
2016
-
[21]
Oore, S.; Simon, I.; Dieleman, S.; Eck, D.; and Simonyan, K. 2020. This time with feeling: learning expressive musical performance. Neural Computing and Applications, 32(4): 955--967
2020
-
[22]
J.; Salamon, J.; Nieto, O.; Liang, D.; and Ellis, D
Raffel, C.; McFee, B.; Humphrey, E. J.; Salamon, J.; Nieto, O.; Liang, D.; and Ellis, D. P. W. 2014. A TRANSPARENT IMPLEMENTATION OF COMMON MIR METRICS . The International Society for Music Information Retrieval (ISMIR)
2014
-
[23]
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text -to- Text Transformer . Journal of Machine Learning Research
2020
-
[24]
Sakoe, H.; and Chiba, S. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1): 43--49
1978
-
[25]
J.; Price, A.; and Taylor, J
Tibshirani, R. J.; Price, A.; and Taylor, J. 2011. A statistician Plays Darts . Journal of the Royal Statistical Society Series A: Statistics in Society, 174(1): 213--226
2011
-
[26]
T.; Landy, M
Trommershäuser, J.; Gepshtein, S.; Maloney, L. T.; Landy, M. S.; and Banks, M. S. 2005. Optimal Compensation for Changes in Task - Relevant Movement Variability . The Journal of Neuroscience, 25(31): 7169--7178
2005
-
[27]
N.; Kaiser, .; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is All you Need . In Advances in Neural Information Processing Systems 30 ( NIPS 2017)
2017
-
[28]
Wang, S.; Ewert, S.; and Dixon, S. 2017. Identifying Missing and Extra Notes in Piano Recordings Using Score - Informed Dictionary Learning . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10): 1877--1889
2017
-
[29]
Wu, Y.; Gardner, J.; Manilow, E.; Simon, I.; Hawthorne, C.; and Engel, J. 2022 a . The Chamber Ensemble Generator : Limitless High - Quality MIR Data via Generative Modeling . ArXiv:2209.14458 [cs, eess]
2022 arXiv
-
[30]
A.; and Engel, J
Wu, Y.; Manilow, E.; Deng, Y.; Swavely, R.; Kastner, K.; Cooijmans, T.; Courville, A.; Huang, C.-Z. A.; and Engel, J. 2022 b . MIDI - DDSP : Detailed Control of Musical Performance via Hierarchical Modeling . In Proceedings of the International Conference on Learning Representations
2022
-
[31]
Ying, X. 2019. An Overview of Overfitting and its Solutions . Journal of Physics: Conference Series, 1168: 022022
2019
-
[32]
Yousician. 2024. Yousician
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.