REVIEW 4 major objections 5 minor 21 references
Optical Music Recognition of Jazz Lead Sheets
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Handwritten jazz lead sheets—melody plus chord symbols—can be transcribed automatically by a region-level encoder-decoder model pretrained on piano OMR and trained with a medium-grained tokeniser, reaching an 11.90% word error rate on held-
desk verdict Solid dataset contribution with credible results, but the split documentation has a potential leakage ambiguity that needs clearing before the numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the medium-level tokeniser: it turns each **kern line into a sequence in which every token corresponds to exactly one musically meaningful graphical symbol, keeping the vocabulary at 153 tokens without forcing character-level recombination. Around it sit a convolutional image encoder with a transformer decoder, pretrained weights from a polyphonic piano OMR task, and synthetic staff images rendered in a handwriting-like font. The tokeniser is what makes chord-note alignment learnable despite the vertical misalignment between handwritten chords and notes.
What would settle it
Transcribe a new set of handwritten lead sheets produced without printed reference copies, with equivalent chord spellings and free layout, and compare against an independent human transcription. If word or line error rates move well above 11.90/29.68, or chord-note alignment fails, the model's generalisation is limited to the dataset's layout-matched, reference-copy setup.
Extended reading notes
Core claim
The paper's central claim is that chord symbols—the component that distinguishes lead sheets from other OMR inputs—can be learned from handwritten images when the transcription is posed at the right granularity. The authors argue that a word-level tokeniser forces the model to memorise long strings, while a character-level tokeniser splits a single graphical symbol across several tokens; the medium choice, where notes, accidentals, chord roots, chord types, extensions, and basses each get their own token, gives the network a clean image-to-token mapping. The experiments show this medium tokeniser beats the other two, that synthetic handwritten-style images help, and that removing the pretrai
Load-bearing premise
The digitised ground-truth scores were made by adapting printed reference files to match each handwritten copy's layout, not by independently reading the handwriting, so the reported error rates assume the handwritten copies are musically identical to those adapted references.
Editorial extensions
If this is right
- The released dataset, ground truth, synthetic images, and split give the community a common benchmark for handwritten lead-sheet OMR.
- A practical pipeline becomes possible: detect staff regions in a scanned fake book and transcribe each region with this model.
- Chord transcription enters OMR: roots, types, extensions, and basses can be recognised and aligned to melody notes even in dirty handwriting.
- Pretraining on a related OMR task is decisive, pointing toward shared cross-domain OMR pretraining for future systems.
- At 29.68 line error rate, roughly one staff line in three still contains at least one character-level mistake, so the result is a working baseline, not a finished transcription tool.
Reading between the lines
- Because the ground-truth files were adapted from printed reference scores to match each handwritten copy's layout instead of being read independently from the handwriting, the reported numbers are best read as upper bounds for this controlled copying setup; free-form handwritten lead sheets could be harder.
- The equivalence-class handling of chord spellings implies the model outputs a semantic chord class rather than the scribe's exact glyphs, so a user-facing renderer could choose between spellings like maj7 and Δ7 without retraining.
- The one-token-per-graphical-symbol principle should transfer to other notations where characters and musical meaning diverge, such as figured bass or tablature.
- The paper's own speculation that larger data favour character-level tokenisation suggests the medium-level advantage is a low-data regime effect, not an asymptotic one.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new OMR dataset of 293 handwritten jazz lead sheets (163 unique pieces) with aligned MusicXML and Humdrum **kern ground truth, plus 326 synthetic renderings, and proposes a region-level encoder-decoder OMR model based on the Sheet Music Transformer architecture. The authors evaluate three tokenization strategies (word, medium, character), the effect of pretraining on polyphonic piano data, and the use of synthetic data. Their best model reaches WER 11.90 / CER 13.67 / LER 29.68 on a held-out test set (Table 1, Section 5.2), and they claim this is the first OMR system to handle chord symbols in handwritten lead sheets.
Significance. If the reported results hold, this is a useful contribution: it provides a much-needed public dataset for handwritten jazz lead sheets with aligned symbolic ground truth, and a reproducible baseline that extends OMR beyond melody-only and polyphonic-piano settings to chord symbols. The paper is notable for releasing code, data, and models, and for a careful ablation of tokenization, pretraining, and synthetic augmentation. The core direction—medium-level tokenization plus pretraining plus synthetic data—is plausible and the qualitative example supports the quantitative findings. However, the paper's evaluation and split description contain ambiguities and inconsistencies that need to be resolved before the central claims are fully trustworthy.
major comments (4)
- [§3.6] The split counts are internally inconsistent. The text assigns 115, 16, and 32 unique pieces to train/validation/test, but then reports 245, 32, and 16 handwritten scores and 1696, 102, and 220 regions for the same three subsets. Literally read, validation has 32 scores for 16 pieces and test has 16 scores for 32 pieces, which contradicts the statement that multi-copy pieces are train-only and makes the test set size ambiguous. The region counts suggest the score counts should be train=245, val=16, test=32 (or the numbers are ordered train/test/val). Please correct this and state exactly how many unique pieces and handwritten scores are in the test set used for Table 1.
- [§3.2/§3.6] The synthetic data split is not explicitly described. Section 3.2 says synthetic images are generated 'for each piece in our digitised scores collection', and Section 3.6 only says that unique pieces with multiple handwritten copies are placed in train. A reader cannot rule out that synthetic renderings of test/validation pieces were included in the training set, which would leak the exact ground-truth token sequences and inflate the reported WER/CER/LER. State explicitly that synthetic images are partitioned by unique piece and that no synthetic image of a test-piece ground truth appears in training.
- [§3.4/§3.5/§4.3] The handling of equivalent chord symbols is under-specified. Section 3.4 says the system maps an image to an equivalence class of labels, and Section 3.5 restricts Harte syntax, but Section 4.3 describes plain edit-distance metrics on predicted vs. ground-truth **kern. It is not stated whether equivalent spellings (e.g., 'maj7' vs 'Δ7', 'm' vs 'min') are canonicalized before tokenization and before computing CER/WER/LER, or whether the model is trained to emit a single canonical spelling. This matters because the reported error rates could either penalize valid predictions or be uninterpretable. Please clarify the canonicalization pipeline.
- [§5.2/Table 1] Table 1 reports a single run per configuration, with no error bars or significance tests. The differences between the best configurations are small (e.g., WER 11.90 for medium vs 12.86 for character-level), and with 32 unique test pieces these differences may not be stable. At minimum, state that results are from a single seed and discuss variance; ideally, provide results over multiple seeds or a small significance analysis.
minor comments (5)
- [§2] Typo: 'transcription of melody and aligned lyrics' should be 'melody and aligned chords'.
- [§3.1] The footnote 'Few samples have polyphonic melodies' sits uneasily with the earlier claim that lead sheets consist of 'a monophonic melody'. Please reconcile or qualify this statement.
- [§3.6] The abstract states 2021 total staves but Section 3.6 sums region counts to 2018 (1696+102+220). Verify the total and the individual split counts.
- [§5.2] The phrase 'pretrained + handwritten + synthetic' in Table 1 is abbreviated; consider defining column headers more explicitly in the caption.
- [§5.1] Please report the number of train/validation/test regions actually used in the experiments, since the split counts in Section 3.6 are currently ambiguous.
Circularity Check
Test-set contamination via synthetic images: training may include synthetic renderings of the same 32 test pieces, making reported WER partially forced.
-
fitted input called prediction
[Section 3.2 (synthetic generation) + Section 3.6 (data split) + Table 1 (Section 5.2, 'pretr. + handw. + synth.' column)]
"for each piece in our digitised scores collection, we generated 2 synthetic full-page renderings ... This ensures that we do not have data in different splits that share the same ground truth ... Specifically, out of 163 unique pieces, we assign 115, 16, and 32 to the train, validation, and test subsets."
The paper generates synthetic renderings for every piece, i.e., all 163 unique pieces, and the split assigns 32 unique pieces to the test set. The 'handwritten + synthetic' and 'pretrained + handwritten + synthetic' training conditions (Table 1) add synthetic data to the training set. If the synthetic images of the 32 test pieces were included in training — and the paper never states that they were excluded — then the training set contains the same ground-truth token sequences (the same pieces) as the test set. The model can memorize the musical content from the synthetic rendering and reproduce it when given the handwritten image of the same piece, so the reported WER 11.90 / CER 13.67 / LER 29.68 are not clean generalization numbers. The stated intent ('we do not have data in different s
full rationale
The central quantitative claim — WER 11.90 / CER 13.67 / LER 29.68 for the best model (pretrained, with synthetic data, medium-level tokeniser) — depends on a clean train/test split. Section 3.2 states synthetic images are generated for each piece in the digitised collection (all 163 unique pieces), while Section 3.6 places 32 unique pieces in the test set and only explicitly guarantees that multiple handwritten copies stay in train. Since the experimental conditions labeled 'handwritten + synthetic' and 'pretr. + handw. + synth.' add synthetic data to training, the text as written permits synthetic renderings of the 32 test pieces to be in the training set. If so, the test labels are present in training via synthetic images, making the reported results partially a measure of memorization rather than generalization. This is a fitted-input-called-prediction situation: the model is fitted on data sharing ground truth with the test set, then 'predicts' that test set. No other circularity is present: the architecture and pretrained checkpoint come from external prior work (Sheet Music Transformer), the tokenisers are independently developed and compared, and the evaluation metrics are standard edit-distance measures. The ground-truth adaptation described in Section 3.4 is a dataset annotation practice rather than a reasoning loop, though it introduces some dependency between labels and images. The score reflects the partial circularity of the evaluation if the synthetic leak exists, as the paper's own split description does not rule it out.
Assumptions & free parameters
assumptions (4)
- domain assumption The digitised scores, after adaptation, are a correct and sufficient transcription of the handwritten lead sheets.
- domain assumption The region bounding boxes and line segmentation (YOLOv8-based detector plus manual review, splitting at linebreak tokens) provide exact alignment between image regions and **kern segments.
- domain assumption Equivalent chord symbols (e.g. maj7 and Delta7) can be treated as a single recognition class, and the Harte-based restricted syntax covers all chords in the collection.
- domain assumption Pretrained weights from the Sheet Music Transformer trained on polyphonic piano music transfer to handwritten jazz lead sheets.
Cite this review
Pith. "Pith review of Optical Music Recognition of Jazz Lead Sheets." pith.science (2026). https://pith.science/paper/EPZMGZ2T
@misc{pith2026250905329,
author = {Pith},
title = {Pith review of: Optical Music Recognition of Jazz Lead Sheets},
year = {2026},
howpublished = {\url{https://pith.science/paper/EPZMGZ2T}},
note = {Machine review of arXiv:2509.05329}
}
read the original abstract
In this paper, we address the challenge of Optical Music Recognition (OMR) for handwritten jazz lead sheets, a widely used musical score type that encodes melody and chords. The task is challenging due to the presence of chords, a score component not handled by existing OMR systems, and the high variability and quality issues associated with handwritten images. Our contribution is two-fold. We present a novel dataset consisting of 293 handwritten jazz lead sheets of 163 unique pieces, amounting to 2021 total staves aligned with Humdrum **kern and MusicXML ground truth scores. We also supply synthetic score images generated from the ground truth. The second contribution is the development of an OMR model for jazz lead sheets. We discuss specific tokenisation choices related to our kind of data, and the advantages of using synthetic scores and pretrained models. We publicly release all code, data, and models.
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION A lead sheet is a kind of sheet music (musical score) that encodes the melody, chords, and sometimes lyrics of a mu- sic composition. Opposed to music styles, such as classical music, where the composer specifies with a high degree of precision what musicians have to play, lead sheets are popular in contexts where a lot of freedom is given to...
arXiv 2025
-
[2]
RELA TED WORK Our problem of transcribing melody and aligned lyrics is very similar to the transcription of melody and lyrics tar- geted by Martinez-Sevilla et al. [3]. They develop a network that, given a music region rotated by 90 degrees it performs an internal reshaping of the hidden space, to horizontally slice the score at note position. The result ...
-
[3]
DA TASET The dataset we release consists of images of musical scores, aligned with a digitised version. Every score encodes a lead sheet of a jazz standard, consisting of a monophonic melody 2 and chord symbols. 3.1 Digitised musical scores We provide musical scores for 163 unique jazz standards in MusicXML and Humdrum **kern format. The latter is widely ...
work page 2013
-
[4]
MODEL In this section, we describe our approach to OMR of jazz lead sheets. We focus on the region-level approach, i.e., our system’s input is a single staff with chords on top. 7 For- mally, each sample in our dataset consists of a pair (x, y) of a region-level image represented as matrix x ∈ Rc×h×w, and a sequence of musical symbols y = y1, y2, . . . , ...
-
[5]
EXPERIMENTS With the experiments in this section, we want to answer the following three research questions: Do pretrained SOTA weights on page-level polyphonic piano music [5] improve the performance in our tasks? Which of the three tokenisers is more effective? Does the inclusion of synthetic data during training help the performance on real data? All co...
-
[6]
CONCLUSION With this paper, we took an initial yet significant step to- wards the development of OMR systems for handwritten jazz lead sheets that could benefit musicians and MIR re- searchers. We collected a dataset of 293 handwritten lead sheets produced by music professionals and students of different levels, addressed its quality problems, aligned it ...
-
[7]
ACKNOWLEDGMENTS We thank ‘Centro Superior de Música del País Vasco, Musikene’ (Itziar Larrinaga), ‘Casa Sofía de El Altet, Ali- cante’ (Pedro J. Ponce de León), ‘Conservatorio Superior de Música Joaquín Rodrigo de Valencia’ (Jorge Sevilla), ‘Sedajazz, Valencia‘ (Francisco A. Blanco Latino), ‘Con- servatorio Superior de Música Óscar Esplá de Alicante’ (Man...
work page 2024
-
[8]
Filosax: A dataset of annotated jazz saxophone recordings,
D. Foster and S. Dixon, “Filosax: A dataset of annotated jazz saxophone recordings,” in Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Srinivasamurthy, Eds., 2021, pp. 205–212
work page 2021
Show all 21 references
-
[9]
Under- standing Optical Music Recognition,
J. Calvo-Zaragoza, J. Hajiˇc Jr., and A. Pacha, “Under- standing Optical Music Recognition,” ACM Comput. Surv., vol. 53, no. 4, pp. 77:1–77:35, 2021
2021
-
[10]
A holistic approach for aligned music and lyrics transcription,
J. C. Martinez-Sevilla, A. Rios-Vila, F. J. Castellanos, and J. Calvo-Zaragoza, “A holistic approach for aligned music and lyrics transcription,” in International Confer- ence on Document Analysis and Recognition (ICDAR), 2023
2023
-
[11]
End-to-end optical music recognition for pianoform sheet music,
A. Ríos-Vila, D. Rizo, J. M. Iñesta, and J. Calvo- Zaragoza, “End-to-end optical music recognition for pianoform sheet music,” International Journal on Doc- ument Analysis and Recognition (IJDAR), vol. 26, no. 3, pp. 347–362, 2023
2023
-
[12]
Sheet music transformer: End-to-end optical music recogni- tion beyond monophonic transcription,
A. Ríos-Vila, J. Calvo-Zaragoza, and T. Paquet, “Sheet music transformer: End-to-end optical music recogni- tion beyond monophonic transcription,” inInternational Conference on Document Analysis and Recognition (IC- DAR), 2024
2024
-
[13]
Humdrum and Kern: Selective Feature En- coding BT - Beyond MIDI: The handbook of musi- cal codes,
D. Huron, “Humdrum and Kern: Selective Feature En- coding BT - Beyond MIDI: The handbook of musi- cal codes,” in Beyond MIDI: The handbook of musical codes. Cambridge, MA, USA: MIT Press, jan 1997, pp. 375–401
1997
-
[14]
The coordinated cor- pus of popular musics (cocopops): A meta-corpus of melodic and harmonic transcriptions
C. Arthur and N. Condit-Schultz, “The coordinated cor- pus of popular musics (cocopops): A meta-corpus of melodic and harmonic transcriptions.” in Proceedings of the International Society for Music Information Re- trieval Conference (ISMIR), 2023
2023
-
[15]
Symbolic representation of musical chords: A proposed syntax for text annotations
C. Harte, M. B. Sandler, S. A. Abdallah, and E. Gómez, “Symbolic representation of musical chords: A proposed syntax for text annotations.” in Proceedings of the Inter- national Society for Music Information Retrieval Con- ference (ISMIR), 2005
2005
-
[16]
Choco: a chord corpus and a data trans- formation workflow for musical harmony knowledge graphs,
J. de Berardinis, A. Meroño-Peñuela, A. Poltronieri, and V . Presutti, “Choco: a chord corpus and a data trans- formation workflow for musical harmony knowledge graphs,” Scientific Data, vol. 10, no. 1, p. 641, 2023
2023
-
[17]
Ultralytics yolov8,
G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics yolov8,” 2023. [Online]. Available: https://github.com/ ultralytics/ultralytics
2023
-
[18]
Staff layout anal- ysis using the YOLO platform,
V . Dvorák, J. Hajiˇc Jr., and J. Mayer, “Staff layout anal- ysis using the YOLO platform,” in 6 th International Workshop on Reading Music Systems, 2024, p. 18
2024
-
[19]
A convnet for the 2020s,
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022
2022
-
[20]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Infor- mation Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar...
2017
-
[21]
MidiTok: A python package for MIDI file tokenization,
N. Fradet, J.-P. Briot, F. Chhel, A. El Fal- lah Seghrouchni, and N. Gutowski, “MidiTok: A python package for MIDI file tokenization,” in Extended Ab- stracts for the Late-Breaking Demo Session of the Inter- national Society for Music Information Retrieval Con- ference, 2021
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.