REVIEW 4 major objections 4 minor 1 cited by
Beat and Downbeat Tracking in Performance MIDI Using an End-to-End Transformer Architecture
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that a transformer encoder-decoder trained on tokenized performance MIDI can track beats and downbeats more accurately than existing symbolic-music systems on most tested datasets.
desk verdict A useful T5-based symbolic beat tracker with honest ablations, but the SOTA claim rests on an under-specified baseline comparison that needs tightening before publication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a T5 encoder-decoder transformer treating beat tracking as a translation from a MIDI language to a rhythm language. Inputs are segments of performance MIDI tokenized with absolute time tokens quantized to fixed steps (10 ms in ablations, 50 ms in the final model), plus tokens for note onsets, note offsets, and velocity; targets are beat tokens, with or without an explicit beat-counter value, placed at quantized times. The pipeline splits MIDI into overlapping 10-second segments with a 1-second hop, cleans segments with fewer than one beat, and applies dynamic augmentation: pitch transposition, random time shift within one second, and time scaling between 0.9 and 1.1. Inference uses beam search with five beams and a no-repeated-bigram constraint. This machinery is what lets the model capture temporal dependencies and output beat positions directly instead of using a post-processing dynamic Bayesian network.
What would settle it
Run the published HMM and PM2S systems on the exact same overlapping 10-second segments, same token quantization, same inference aggregation, and same 70 ms tolerance used here; if their F1 scores match or exceed the reported ones on A-MAPS and GuitarSet, the central claim of superiority would be refuted. A second check is to evaluate the final 50 ms-quantized model with the tolerance widened or narrowed to see whether the downbeat gains survive at higher temporal precision.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a T5-style encoder-decoder transformer trained from scratch on overlapping 10-second MIDI segments, with time values quantized to 10-50 ms and encoded as tokens for note onsets, offsets, and optionally velocity, can translate MIDI into beat and downbeat annotations with higher F1 than prior MIDI-only approaches. The strongest reported result is on A-MAPS, where beat F1 reaches 98.01 percent and downbeat F1 76.56 percent, versus 83.89 percent and 68.90 percent for the PM2S baseline and below 50 percent for the HMM baselines. On ASAP the model's beat F1 trails PM2S, but its downbeat F1 is roughly double. The authors attribute the gains to the translation formulation, velocity and offset tokens, dynamic pitch and time augmentation, and to quantization choices that interact with the 70 ms evaluation tolerance.
Load-bearing premise
The load-bearing premise is that the HMM and PM2S baselines were trained and evaluated under the same protocol as the proposed model (same segment extraction, same quantization, same 70 ms tolerance, same treatment of the first five seconds); the paper does not describe the baseline configurations, so an uneven comparison could explain the reported gains.
Editorial extensions
If this is right
- If the claim holds, MIDI-only beat tracking can be treated as text translation, so advances in sequence-to-sequence models transfer directly to the task.
- Velocity and note-offset information are worth keeping when available: including them raises beat F1 by roughly 18 percentage points over onset-only input.
- Removing explicit beat-counter values from the target sequence substantially improves downbeat F1, suggesting the model should infer bar-relative positions implicitly.
- Coarser time quantization up to about 100 ms inflates reported F1 under the 70 ms tolerance, so evaluation protocols must state quantization step and tolerance together.
- The trained model generalizes across piano and guitar datasets despite instrument-specific annotation quality differences.
Reading between the lines
- If the method is combined with an audio-to-MIDI transcription front end, the resulting pipeline could give a fair audio-input comparison against audio-based beat trackers, something the paper itself suggests.
- The finding that velocity tokens help downbeat detection implies that any future MIDI source lacking velocity, such as quantized score MIDI, will likely need alternative accent cues or a generative model that imputes them.
- A testable extension would be to train the same tokenizer and architecture on tatum or subdivision labels, since the translation framework likely extends to finer metrical levels.
- Because the reported superiority rests on protocol consistency with baselines, re-running the published baselines under the paper's exact segment and quantization settings is the fastest way to verify the gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an end-to-end transformer-based method for beat and downbeat tracking from performance MIDI. The model is a reduced T5 encoder-decoder trained to translate tokenized MIDI segments into tokenized beat sequences. The authors propose several MIDI-specific tokenization schemes, dynamic augmentation (pitch transposition, time shift, time scaling), and report ablations over encoding variants, segment length, model architecture, augmentation, and time quantization on the A-MAPS dataset. The final model is trained on combined train splits of A-MAPS, ASAP, GuitarSet, and Leduc, and evaluated against an HMM-based method [12] and PM2S [13] on test splits of those four datasets. The paper claims state-of-the-art symbolic MIDI beat tracking, outperforming the baselines on most datasets for both beat and downbeat F1.
Significance. If the central comparative claim is supported, the paper would make a useful contribution: it demonstrates that a sequence-to-sequence transformer can perform symbolic beat tracking across piano and guitar datasets, and it releases beat/downbeat annotations for the Leduc dataset, which is a concrete resource for the community. The ablation study is broad and the experimental scope across four datasets is commendable. However, the state-of-the-art claim is not yet established because the baseline comparison is uncontrolled, hyperparameters were selected on the test split, and no error bars or significance tests are reported. These issues are load-bearing for the paper's main claim, so the contribution is currently more of a promising feasibility study than a validated SOTA result.
major comments (4)
- [Section 4.2, Table 7] The central comparative claim that the proposed model outperforms PM2S and the HMM is not supported by the information given about how the baselines were configured. The paper specifies the proposed model's protocol (70 ms tolerance, no exclusion of the first five seconds, 10-second segments with a 1-second hop) in Section 3.2, but it does not state whether PM2S and the HMM were evaluated with the same train/test splits, the same segment-wise inference procedure, the same tolerance, or the same first-five-seconds policy. Since PM2S is a full-piece CRNN and the HMM is a global probabilistic model, their results are likely sensitive to these choices; the margins in Table 7 could therefore reflect protocol differences rather than genuine model superiority.
- [Section 4.1, Tables 2-6; Section 4.2] The final model configuration was selected using the A-MAPS test split: the ablation tables report F1 scores computed on the test split, and Section 4.2 states that the final model uses the segment length, time quantization, and augmentation combination found best in the ablation study. No held-out validation set, repeated runs, error bars, or significance tests are reported. This makes it hard to judge whether the differences in Table 7 (for example, PM2S 82.95% vs. ours 78.13% beat F1 on ASAP, or 42.63% vs. 52.38% on GuitarSet) are meaningful, especially because several ablation gaps are small relative to the likely run-to-run variance of a randomly initialized transformer.
- [Section 3.1, Leduc description; Section 4.2, Table 7] The Leduc beat and downbeat annotations used as ground truth in Table 7 were created by the authors through an alignment process that is described as using the approach of Riley et al. and then 'further refined' to adjust beat and downbeat information. The paper provides no evaluation of the reliability of these annotations and no comparison with independent annotations. Since Leduc is one of the four datasets supporting the state-of-the-art claim, the accuracy of these annotations is load-bearing and needs to be documented, for example by reporting agreement with a second annotator or with an established alignment method.
- [Section 3.2, Section 4.2] The procedure for converting the model's per-segment outputs into full-piece predictions is not described. The model is trained and evaluated on 10-second segments with a 1-second hop, but Table 7 reports dataset-level F1 scores. The paper does not say how overlapping segment predictions are aggregated (e.g., taking the prediction of the first segment covering each time point, voting, or postprocessing), nor how pieces shorter or longer than the segment length are handled. Without this information the reported F1 numbers are not reproducible and the comparison with full-piece baselines remains ambiguous.
minor comments (4)
- [Section 3.1] There is a typo: 'orginal Leduc dataset' should be 'original Leduc dataset'.
- [Section 4.2, Table 7] The sentence 'we can see a significant drop in downbeat accuracy when comparing the results for the A-MAPS dataset with the others' is confusing because Table 7 shows A-MAPS with the highest downbeat F1 (76.56%), not a drop. Please rephrase to say that downbeat accuracy is lower on the other datasets than on A-MAPS.
- [Section 4.1.1] The statement that adding velocity and offset information 'leads to an 18% higher fb score' compares v3 (96.03%) with v1 (81.06%); this is a relative increase of about 18%, but an absolute increase of about 15 percentage points. Please state explicitly whether the percentage is relative or absolute.
- [Section 4.2, Table 7] The Beat This! results are explicitly labeled as 'intended to give an impression rather than a fair comparison,' yet they appear in the same comparative table as the MIDI baselines. Please move these numbers to a clearly separated reference block or remove them from the main comparison table to avoid misleading readers.
Circularity Check
No significant circularity: the paper is a supervised-learning benchmark with no derivation chain that reduces to its own inputs.
full rationale
The paper's central claim is empirical: a T5 encoder-decoder is trained on tokenized MIDI segments with beat/downbeat targets and compared against HMM and PM2S baselines on test splits (Table 7). There is no equation-to-equation reduction, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from prior work. The Leduc beat annotations were created by the authors via alignment of GuitarPro scores with performances (Section 3.1), but these annotations are used as ground truth for all methods equally; training and evaluation use separate split portions of the same annotation source, which is ordinary supervised learning rather than circularity. The final configuration was chosen after reviewing ablation results obtained on the A-MAPS test split (Sections 4.1 and 4.2), and the baseline protocol for PM2S/HMM is under-specified; both are methodological or protocol concerns affecting validity and fairness, not circularity of the kind defined here. The paper explicitly disclaims the Beat This! comparison as not fair (Section 4.2), and explicitly notes the 70 ms tolerance caveat (Section 4.1.5). No self-citation is load-bearing; no ansatz is smuggled in via citation. Hence no circular steps are identified.
Assumptions & free parameters
free parameters (8)
- Segment length =
10 s
- Time quantization step =
50 ms
- Augmentation ranges =
pitch transpose A0-C8, time shift +/-1 s, time scale 0.9-1.1
- Beam size =
5
- Model dimensions =
d_model=128, d_ff=1024, 3 layers, 4 heads
- N-gram block =
2
- Training epochs =
50
- Batch size =
32
assumptions (6)
- domain assumption MIDI note events contain sufficient timing and velocity information to infer beat and downbeat positions.
- domain assumption The A-MAPS, ASAP, GuitarSet, and Leduc beat annotations are accurate ground truth.
- ad hoc to paper The Leduc beat and downbeat annotations created by the authors via alignment are reliable enough to serve as evaluation ground truth.
- domain assumption Segment-wise training on 10 s windows transfers to full-piece evaluation.
- standard math mir_eval F1 with a 70 ms tolerance window is the appropriate evaluation metric.
- ad hoc to paper The HMM and PM2S baselines are directly comparable without retraining or protocol matching.
Cite this review
Pith. "Pith review of Beat and Downbeat Tracking in Performance MIDI Using an End-to-End Transformer Architecture." pith.science (2026). https://pith.science/paper/K74YCJ5X
@misc{pith2026250700466,
author = {Pith},
title = {Pith review of: Beat and Downbeat Tracking in Performance MIDI Using an End-to-End Transformer Architecture},
year = {2026},
howpublished = {\url{https://pith.science/paper/K74YCJ5X}},
note = {Machine review of arXiv:2507.00466}
}
read the original abstract
Beat tracking in musical performance MIDI is a challenging and important task for notation-level music transcription and rhythmical analysis, yet existing methods primarily focus on audio-based approaches. This paper proposes an end-to-end transformer-based model for beat and downbeat tracking in performance MIDI, leveraging an encoder-decoder architecture for sequence-to-sequence translation of MIDI input to beat annotations. Our approach introduces novel data preprocessing techniques, including dynamic augmentation and optimized tokenization strategies, to improve accuracy and generalizability across different datasets. We conduct extensive experiments using the A-MAPS, ASAP, GuitarSet, and Leduc datasets, comparing our model against state-of-the-art hidden Markov models (HMMs) and deep learning-based beat tracking methods. The results demonstrate that our model outperforms existing symbolic music beat tracking approaches, achieving competitive F1-scores across various musical styles and instruments. Our findings highlight the potential of transformer architectures for symbolic beat tracking and suggest future integration with automatic music transcription systems for enhanced music analysis and score generation.
Figures
Forward citations
Cited by 1 Pith paper
-
Masked diffusion enables coherent beat tracking
Masked diffusion with independent beat/downbeat masking, a balanced unmasking schedule, and inter-step peak picking reduces incoherent beat-tracking outputs and improves GTZAN scores.
Reference graph
Works this paper leans on
-
[12]
Joint Beat and Downbeat Tracking with Recurrent Neural Networks,
S. Böck, F. Krebs, and G. Widmer, “Joint Beat and Downbeat Tracking with Recurrent Neural Networks,” in Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR), 2016
work page 2016
-
[13]
Temporal Convolutional Networks for Musical Audio Beat Tracking,
M. E. P. Davies and S. Böck, “Temporal Convolutional Networks for Musical Audio Beat Tracking,” in 27th European Signal Processing Conference (EUSIPCO), 2019
work page 2019
-
[1]
This rhythmic grid con- sists of downbeats, beats, and tatum subdivisions
INTRODUCTION Beat tracking aims to detect the underlying rhythmic grid within a musical performance [1]. This rhythmic grid con- sists of downbeats, beats, and tatum subdivisions. Down- beats refer to the first beat within a bar and therefore indi- cate the beginning of a new bar. In a notation-level music transcription system, we implicitly or explicitly...
arXiv 2011
-
[2]
METHODOLOGY Our proposed approach for performance MIDI beat tracking is visualized in Figure 1. The model follows an encoder- decoder transformer architecture, designed to translate an input MIDI segment into the corresponding beat sequence. Since transformers operate on text-based token sequences, MIDI data must first be preprocessed and tokenized before...
-
[3]
As input, we use the MIDI files of the A-MAPS dataset. Firstly, we extract the notes and the beat annotations from the MIDI files using the PrettyMIDI library [17]. The ex- tracted annotations are then split into segments of 10 s with a hop size of 1 s. In the next step, the segments are cleaned, and all examples with less than one beat are dropped. Fi- n...
-
[4]
EXPERIMENTS In this section, we describe the datasets used for training and evaluation, followed by the metrics used to assess model performance. 3.1 Datasets For the training and evaluation of the model, datasets con- taining synchronized MIDI and beat annotations are essen- tial. The A-MAPS dataset [21] is an extension of the MAPS database [22] which co...
-
[5]
RESULTS This section presents the results of our proposed method, evaluated using the experimental setup described in Section
-
[6]
We first analyze the impact of various hyperparameters and modeling choices in an ablation study, followed by a comparative evaluation against existing state-of-the-art approaches. 4.1 Ablation Study In the ablation study, we show the effect of various hyperpa- rameters, encoding schemes, and data augmentation strate- gies. Each model is trained for 50 ep...
Show all 36 references
-
[7]
Model Architecture 𝑓b 𝑓db T5 96.03 % 59.52 % GPT2 88.69 % 45.22 % Table 4
We can see that the T5 clearly outperforms the GPT2 model in terms of beat and downbeat F1-scores. Model Architecture 𝑓b 𝑓db T5 96.03 % 59.52 % GPT2 88.69 % 45.22 % Table 4. Comparison of beat ( 𝑓b) and downbeat (𝑓db) F1- scores for different NLP task interpretations. The resu...
-
[8]
CONCLUSION This research demonstrates the effectiveness of an end- to-end transformer-based approach for beat tracking in MIDI performances. By formulating the task as a symbolic translation problem, our model surpasses existing MIDI- based methods, including HMM-based approac...
-
[9]
M. E. P. Davies, S. Böck, and M. Fuentes, Tempo, Beat and Downbeat Estimation . https://tempobeatdownbeat.github.io/tutorial/intro.html, 2021
2021
-
[10]
Auto- matic Music Transcription: An Overview,
E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Auto- matic Music Transcription: An Overview,”IEEE Signal Processing Magazine, vol. 36, no. 1, pp. 20–30, 2018
2018
-
[11]
Enhanced Beat Tracking with Context-Aware Neural Networks,
S. Böck and M. Schedl, “Enhanced Beat Tracking with Context-Aware Neural Networks,” inProceedings of the 14th International Conference on Digital Audio Effects (DAFx), 2011
2011
-
[14]
Beat Transformer: Demixed Beat and Downbeat Tracking with Dilated Self-Attention,
J. Zhao, G. Xia, and Y . Wang, “Beat Transformer: Demixed Beat and Downbeat Tracking with Dilated Self-Attention,” in Proceedings of the 23rd Interna- tional Society for Music Information Retrieval Confer- ence (ISMIR), 2022
2022
-
[15]
Beat This! Accurate beat tracking without DBN postprocessing,
F. Foscarin, J. Schlüter, and G. Widmer, “Beat This! Accurate beat tracking without DBN postprocessing,” in Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024
2024
-
[16]
From MIDI to Traditional Mu- sical Notation,
E. Cambouropoulos, “From MIDI to Traditional Mu- sical Notation,” in Proceedings of the AAAI Workshop on Artificial Intelligence and Music: Towards Formal Models for Composition, Performance and Analysis , 2000
2000
-
[17]
Temperley,Music and Probability
D. Temperley,Music and Probability. Mit Press, 2007
2007
-
[18]
Transcribing Human Piano Performances Into Music Notation,
A. Cogliati, D. Temperley, and Z. Duan, “Transcribing Human Piano Performances Into Music Notation,” in Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR), 2016
2016
-
[19]
A Parse-Based Framework for Coupled Rhythm Quan- tization and Score Structuring,
F. Foscarin, F. Jacquemard, P. Rigaux, and M. Sakai, “A Parse-Based Framework for Coupled Rhythm Quan- tization and Score Structuring,” in Mathematics and Computation in Music, 2019
2019
-
[20]
Non-Local Musical Statistics As Guides for Audio-To-Score Piano Transcription,
K. Shibata, E. Nakamura, and K. Yoshii, “Non-Local Musical Statistics As Guides for Audio-To-Score Piano Transcription,” Information Sciences, vol. 566, pp. 262– 280, 2021
2021
-
[21]
Per- formance MIDI-To-Score Conversion by Neural Beat Tracking,
L. Liu, Q. Kong, G. Morfi, E. Benetos et al. , “Per- formance MIDI-To-Score Conversion by Neural Beat Tracking,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference (IS- MIR), 2022
2022
-
[22]
Note-Level Auto- matic Guitar Transcription Using Attention Mechanism,
S. Kim, T. Hayashi, and T. Toda, “Note-Level Auto- matic Guitar Transcription Using Attention Mechanism,” in Proceedings of the 30th European Signal Processing Conference (EUSIPCO), 2022
2022
-
[23]
End-to-End Piano Performance- MIDI To Score Conversion with Transformers,
T. Beyer and A. Dai, “End-to-End Piano Performance- MIDI To Score Conversion with Transformers,” inPro- ceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024
2024
-
[24]
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020
2020
-
[25]
Intuitive Analysis, Creation and Manipulation of MIDI Data with pretty_midi,
C. Raffel and D. P. Ellis, “Intuitive Analysis, Creation and Manipulation of MIDI Data with pretty_midi,” in Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR), 2014
2014
-
[26]
Sequence-to-Sequence Piano Transcription with Transformers,
C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. Engel, “Sequence-to-Sequence Piano Transcription with Transformers,” in Proceedings of the 22nd Interna- tional Society for Music Information Retrieval Confer- ence (ISMIR), 2021
2021
-
[27]
Hugging Face’s Transformers: State-of- the-Art Natural Language Processing,
T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. De- langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz et al., “Hugging Face’s Transformers: State-of- the-Art Natural Language Processing,” arXiv preprint arXiv:1910.03771, 2019
1910 arXiv
-
[28]
Adafactor: Adaptive Learn- ing Rates with Sublinear Memory Cost,
N. Shazeer and M. Stern, “Adafactor: Adaptive Learn- ing Rates with Sublinear Memory Cost,” in Interna- tional Conference on Machine Learning. PMLR, 2018, pp. 4596–4604
2018
-
[29]
A-MAPS: Augmented MAPS Dataset with Rhythm and Key Annotations,
A. Ycart and E. Benetos, “A-MAPS: Augmented MAPS Dataset with Rhythm and Key Annotations,” in Pro- ceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), 2018
2018
-
[30]
Multipitch Es- timation of Piano Sounds Using a New Probabilistic Spectral Smoothness Principle,
V . Emiya, R. Badeau, and B. David, “Multipitch Es- timation of Piano Sounds Using a New Probabilistic Spectral Smoothness Principle,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 6, pp. 1643–1654, 2009
2009
-
[31]
ASAP: A Dataset of Aligned Scores and Performances for Piano Transcription,
F. Foscarin, A. Mcleod, P. Rigaux, F. Jacquemard, and M. Sakai, “ASAP: A Dataset of Aligned Scores and Performances for Piano Transcription,” in Proceedings of the 21st International Society for Music Information Retrieval Conference (ISMIR), 2020
2020
-
[32]
GuitarSet: A Dataset for Guitar Transcription,
Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “GuitarSet: A Dataset for Guitar Transcription,” in Pro- ceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), 2018
2018
-
[33]
The François Leduc Dataset,
D. Edwards, X. Riley, and S. Dixon, “The François Leduc Dataset,” Apr. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10984521
2024 doi
-
[34]
High Resolu- tion Guitar Transcription via Domain Adaptation,
X. Riley, D. Edwards, and S. Dixon, “High Resolu- tion Guitar Transcription via Domain Adaptation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024
2024
-
[35]
MIR_EV AL: A Transparent Implementation of Common MIR Met- rics,
C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Ni- eto, D. Liang, D. P. Ellis, and C. C. Raffel, “MIR_EV AL: A Transparent Implementation of Common MIR Met- rics,” in Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR), 2014
2014
-
[36]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.