{"id":"3f804ec5-d44f-4f92-b778-a6e242d1ed4e","arxiv_id":"2509.05329","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new dataset of 293 handwritten jazz lead sheets with digital ground truth, plus an encoder-decoder OMR model that transcribes melody and chords, reaching 11.90 WER.","lead":"Researchers built a dataset of 293 handwritten jazz lead sheets aligned to digital ground truth and trained an optical music recognition model that reads melody and chord symbols from staff images. The best model reaches about 12% word error rate on unseen pieces, and all code, data, and models are released publicly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic images of test pieces may be in the training set, leaking ground truth and inflating reported WER/CER/LER.","rationale":"The reader's weakest_assumption focused on ground-truth adaptation (Section 3.4), which is a valid but less directly testable concern about annotation quality. However, the reader's rationale also flagged the split ambiguity. I identify the synthetic-data leakage as the single most load-bearing concern because it directly determines whether the reported test metrics measure what they claim: generalization to unseen handwritten lead sheets. If synthetic images of test pieces are in training, the headline results are not evidence of OMR capability; they are evidence of memorization. This concern is concrete and checkable from the released dataset, unlike the ground-truth adaptation issue, which is a design limitation that does not necessarily invalidate the comparison (the adapted GT still represents the original musical content, and the chord-equivalence handling addresses the known symbol variations). The paper's own stated principle—'we do not have data in different splits that share the same ground truth'—makes it likely the authors intended to exclude synthetic test pieces, but the text does not say so explicitly. Given that the dataset and code are released, a reviewer can settle this immediately. Therefore, the appropriate verdict is CONDITIONAL: the paper's central empirical claim is accepted only if the split is clean; otherwise the quantitative results must be revised. The reader's CONDITIONAL verdict remains appropriate, but the specific condition should be sharpened to the synthetic-data split, which is more directly tied to the numerical claims than the GT adaptation concern.","tokens_in":10121,"tokens_out":5763,"duration_ms":70205,"concrete_test":"Inspect the released dataset's split metadata or directory structure to determine whether any synthetic image (PNG/SVG or MuseScore rendering) whose piece ID is assigned to the test set appears in the training data (e.g., in the training folder or training manifest). If none, the concern is resolved and the reported metrics stand. If synthetic test-piece images are present in training, retrain the best model (pretrained, medium-level tokeniser, handwritten+synthetic) with those images removed, re-evaluate on the same test set, and compare WER/CER/LER. A substantial degradation would confirm the leak and invalidate the original numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim is that the best model (pretrained, with synthetic data, medium-level tokeniser) reaches WER 11.90 / CER 13.67 / LER 29.68 on a held-out test set (Table 1, Section 5.2). This claim depends critically on the train/test split being clean. Section 3.6 describes the split: 70/10/20 of 163 unique pieces, and states: 'We ensure that all unique pieces for which multiple handwritten copies exist are placed in the train subset only. This ensures that we do not have data in different splits that share the same ground truth.' However, this sentence only mentions handwritten copies. Synthetic images were generated 'for each piece in our digitised scores collection' (Section 3.2), i.e., for all 163 unique pieces, including those assigned to test. If synthetic renderings of test pieces were included in the training set, the model has already seen the exact same piece (rendered with MuseScore) and its ground-truth token sequence during training. It could then memorize that sequence and reproduce it from the handwritten test image, inflating all metrics. The text is ambiguous: the stated principle 'no data sharing ground truth across splits' should also exclude synthetic test pieces, but it is not explicitly stated that this was done. If the leak exists, the reported test scores are not a measure of generalization to unseen lead sheets, undermining the central claim of a functional OMR system for handwritten jazz lead sheets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new OMR dataset of 293 handwritten jazz lead sheets (163 unique pieces) with aligned MusicXML and Humdrum **kern ground truth, plus 326 synthetic renderings, and proposes a region-level encoder-decoder OMR model based on the Sheet Music Transformer architecture. The authors evaluate three tokenization strategies (word, medium, character), the effect of pretraining on polyphonic piano data, and the use of synthetic data. Their best model reaches WER 11.90 / CER 13.67 / LER 29.68 on a held-out test set (Table 1, Section 5.2), and they claim this is the first OMR system to handle chord symbols in handwritten lead sheets.","tokens_in":10429,"tokens_out":9692,"duration_ms":110525,"significance":"If the reported results hold, this is a useful contribution: it provides a much-needed public dataset for handwritten jazz lead sheets with aligned symbolic ground truth, and a reproducible baseline that extends OMR beyond melody-only and polyphonic-piano settings to chord symbols. The paper is notable for releasing code, data, and models, and for a careful ablation of tokenization, pretraining, and synthetic augmentation. The core direction—medium-level tokenization plus pretraining plus synthetic data—is plausible and the qualitative example supports the quantitative findings. However, the paper's evaluation and split description contain ambiguities and inconsistencies that need to be resolved before the central claims are fully trustworthy.","major_comments":[{"comment":"The split counts are internally inconsistent. The text assigns 115, 16, and 32 unique pieces to train/validation/test, but then reports 245, 32, and 16 handwritten scores and 1696, 102, and 220 regions for the same three subsets. Literally read, validation has 32 scores for 16 pieces and test has 16 scores for 32 pieces, which contradicts the statement that multi-copy pieces are train-only and makes the test set size ambiguous. The region counts suggest the score counts should be train=245, val=16, test=32 (or the numbers are ordered train/test/val). Please correct this and state exactly how many unique pieces and handwritten scores are in the test set used for Table 1.","section":"§3.6"},{"comment":"The synthetic data split is not explicitly described. Section 3.2 says synthetic images are generated 'for each piece in our digitised scores collection', and Section 3.6 only says that unique pieces with multiple handwritten copies are placed in train. A reader cannot rule out that synthetic renderings of test/validation pieces were included in the training set, which would leak the exact ground-truth token sequences and inflate the reported WER/CER/LER. State explicitly that synthetic images are partitioned by unique piece and that no synthetic image of a test-piece ground truth appears in training.","section":"§3.2/§3.6"},{"comment":"The handling of equivalent chord symbols is under-specified. Section 3.4 says the system maps an image to an equivalence class of labels, and Section 3.5 restricts Harte syntax, but Section 4.3 describes plain edit-distance metrics on predicted vs. ground-truth **kern. It is not stated whether equivalent spellings (e.g., 'maj7' vs 'Δ7', 'm' vs 'min') are canonicalized before tokenization and before computing CER/WER/LER, or whether the model is trained to emit a single canonical spelling. This matters because the reported error rates could either penalize valid predictions or be uninterpretable. Please clarify the canonicalization pipeline.","section":"§3.4/§3.5/§4.3"},{"comment":"Table 1 reports a single run per configuration, with no error bars or significance tests. The differences between the best configurations are small (e.g., WER 11.90 for medium vs 12.86 for character-level), and with 32 unique test pieces these differences may not be stable. At minimum, state that results are from a single seed and discuss variance; ideally, provide results over multiple seeds or a small significance analysis.","section":"§5.2/Table 1"}],"minor_comments":[{"comment":"Typo: 'transcription of melody and aligned lyrics' should be 'melody and aligned chords'.","section":"§2"},{"comment":"The footnote 'Few samples have polyphonic melodies' sits uneasily with the earlier claim that lead sheets consist of 'a monophonic melody'. Please reconcile or qualify this statement.","section":"§3.1"},{"comment":"The abstract states 2021 total staves but Section 3.6 sums region counts to 2018 (1696+102+220). Verify the total and the individual split counts.","section":"§3.6"},{"comment":"The phrase 'pretrained + handwritten + synthetic' in Table 1 is abbreviated; consider defining column headers more explicitly in the caption.","section":"§5.2"},{"comment":"Please report the number of train/validation/test regions actually used in the experiments, since the split counts in Section 3.6 are currently ambiguous.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the split description: the numerical inconsistencies in Section 3.6 and the unstated treatment of synthetic test-piece renderings. If the authors can provide the exact split logic and confirm that no synthetic image of a test-piece ground truth is in training, the central result is likely salvageable. I would be comfortable with major_revision rather than rejection, because the dataset and ablations are valuable and the problems appear fixable in the text and released metadata."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. 293 handwritten jazz lead sheets with aligned MusicXML and **kern ground truth, plus synthetic renderings, is a genuine gap in OMR, and the release of code and data is commendable. The model work is careful too: the three tokenisation strategies are clearly motivated, and the ablations show that pretraining and synthetic data help, with the medium-level tokeniser winning. The qualitative example suggests the system actually works on dirty handwriting, which is the point of the task.\n\nThe soft spots are in the evaluation, not in the contribution. The biggest one is the split. Section 3.6 says they split 163 unique pieces into 70/10/20 and ensure that pieces with multiple handwritten copies are all in train. But that only addresses handwritten copies; synthetic images were generated for every unique piece, so if synthetic renderings of test pieces were not explicitly excluded from training, the model could memorise the exact token sequence and inflate the reported WER/CER/LER. The text is ambiguous on this, and the counts don't help: test has 32 unique pieces but only 16 handwritten scores, which doesn't add up unless some pieces have no handwritten copy (which they should, per the abstract). This needs a clear statement and a corrected table.\n\nThere are also the standard small-data evaluation gaps: a single run per configuration, no error bars, no baseline comparison against an existing OMR system or a melody-only model, and the ground truth is adapted from the reference score rather than independently transcribed from the handwriting, so the target itself is partly constructed. That said, none of this kills the dataset. The equivalence-class handling of chord symbols is a thoughtful move, though the metrics as reported against exact labels could still penalise correct predictions that use a different but equivalent spelling.\n\nWho is this for? Anyone in music document analysis or MIR who needs a benchmark for lead sheets or chord-symbol recognition. The paper deserves a serious referee, but only after the split ambiguity is resolved and preferably with error bars or a second run. If the split is clean, the results are believable. If not, the model numbers need to be recomputed. Either way, the dataset is worth publishing.","headline":"Solid dataset contribution with credible results, but the split documentation has a potential leakage ambiguity that needs clearing before the numbers can be trusted.","tokens_in":85,"tokens_out":3520,"would_cite":true,"duration_ms":92728,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Handwritten jazz lead sheets—melody plus chord symbols—can be transcribed automatically by a region-level encoder-decoder model pretrained on piano OMR and trained with a medium-grained tokeniser, reaching an 11.90% word error rate on held-","keywords":["Optical Music Recognition","handwritten music scores","jazz lead sheets","chord symbols","tokenisation","encoder-decoder model","sheet music dataset","**kern"],"falsifier":"Transcribe a new set of handwritten lead sheets produced without printed reference copies, with equivalent chord spellings and free layout, and compare against an independent human transcription. If word or line error rates move well above 11.90/29.68, or chord-note alignment fails, the model's generalisation is limited to the dataset's layout-matched, reference-copy setup.","tokens_in":10049,"feed_emoji":"🎼","tokens_out":8908,"duration_ms":103543,"temperature":0.7,"pith_summary":"Optical Music Recognition has handled printed and handwritten scores, but not jazz lead sheets—a score type that puts a melody line and chord symbols together. This paper claims such sheets can be transcribed automatically by a neural model that reads one staff at a time, provided it starts from a checkpoint pretrained on piano-sheet OMR, is trained on both real handwritten and synthetic staff images, and uses a 'medium-level' tokeniser in which each graphical symbol (pitch, accidental, chord root, chord type, extension, bass) maps to exactly one token. To support the claim it releases an open dataset of 293 handwritten lead sheets of 163 pieces, 2021 staves aligned to digitised ground truth, plus synthetic renderings and a benchmark split. On a held-out set of 32 pieces, the best configuration reaches 11.90% word error rate, 13.67% character error rate, and 29.68% line error rate, with the remaining errors concentrated in chord symbols. If correct, this is the first OMR system for handwritten lead sheets and a usable baseline for turning jazz fake books into digital scores.","feed_headline":"First system reads handwritten jazz lead sheets","feed_subtitle":"A new dataset plus model turns melody-and-chord sheets into digital scores, errors mostly on chords.","key_machinery":"The load-bearing mechanism is the medium-level tokeniser: it turns each **kern line into a sequence in which every token corresponds to exactly one musically meaningful graphical symbol, keeping the vocabulary at 153 tokens without forcing character-level recombination. Around it sit a convolutional image encoder with a transformer decoder, pretrained weights from a polyphonic piano OMR task, and synthetic staff images rendered in a handwriting-like font. The tokeniser is what makes chord-note alignment learnable despite the vertical misalignment between handwritten chords and notes.","core_discovery":"The paper's central claim is that chord symbols—the component that distinguishes lead sheets from other OMR inputs—can be learned from handwritten images when the transcription is posed at the right granularity. The authors argue that a word-level tokeniser forces the model to memorise long strings, while a character-level tokeniser splits a single graphical symbol across several tokens; the medium choice, where notes, accidentals, chord roots, chord types, extensions, and basses each get their own token, gives the network a clean image-to-token mapping. The experiments show this medium tokeniser beats the other two, that synthetic handwritten-style images help, and that removing the pretrai","pith_inferences":["Because the ground-truth files were adapted from printed reference scores to match each handwritten copy's layout instead of being read independently from the handwriting, the reported numbers are best read as upper bounds for this controlled copying setup; free-form handwritten lead sheets could be harder.","The equivalence-class handling of chord spellings implies the model outputs a semantic chord class rather than the scribe's exact glyphs, so a user-facing renderer could choose between spellings like maj7 and Δ7 without retraining.","The one-token-per-graphical-symbol principle should transfer to other notations where characters and musical meaning diverge, such as figured bass or tablature.","The paper's own speculation that larger data favour character-level tokenisation suggests the medium-level advantage is a low-data regime effect, not an asymptotic one."],"forward_implications":["The released dataset, ground truth, synthetic images, and split give the community a common benchmark for handwritten lead-sheet OMR.","A practical pipeline becomes possible: detect staff regions in a scanned fake book and transcribe each region with this model.","Chord transcription enters OMR: roots, types, extensions, and basses can be recognised and aligned to melody notes even in dirty handwriting.","Pretraining on a related OMR task is decisive, pointing toward shared cross-domain OMR pretraining for future systems.","At 29.68 line error rate, roughly one staff line in three still contains at least one character-level mistake, so the result is a working baseline, not a finished transcription tool."],"supporting_citations":[{"why":"Supplies the encoder-decoder architecture and the pretrained checkpoint that the experiments show is necessary for effective training.","marker":"[5]"},{"why":"Defines the convolutional encoder that compresses the staff image into a hidden representation.","marker":"[12]"},{"why":"Provides the transformer decoder that autoregressively generates the token sequence from the image representation.","marker":"[13]"},{"why":"Proposes the chord-syntax notation that the paper restricts into its canonical root:shorthand(extensions)/bass form.","marker":"[8]"},{"why":"Defines the **kern text format on which the tokenisers, reconstruction, and evaluation metrics operate.","marker":"[6]"},{"why":"Presents the aligned melody-and-lyrics approach whose vertical-alignment assumption fails on handwritten lead sheets, motivating the chosen architecture.","marker":"[3]"},{"why":"Establishes the polyphonic piano OMR setting from which the pretraining benefits are drawn.","marker":"[4]"},{"why":"Provides the region-detection model used to obtain the staff bounding boxes that define the region-level task.","marker":"[10]"}],"fun_headline_variants":["AI reads handwritten jazz lead sheets with chords","Decoding chord symbols from jazz sheet images","First OMR system for melody and chord charts","Turning scrawled jazz charts into digital scores"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The digitised ground-truth scores were made by adapting printed reference files to match each handwritten copy's layout, not by independently reading the handwriting, so the reported error rates assume the handwritten copies are musically identical to those adapted references.","fun_headline_variants_meta":{"raw":{"variants":["AI reads handwritten jazz lead sheets with chords","Decoding chord symbols from jazz sheet images","First OMR system for melody and chord charts","Turning scrawled jazz charts into digital scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1097,"prompt_tokens":661,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":379}},"tokens_in":405,"tokens_out":436,"duration_ms":6973,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:06:20.502762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Transcribe a new set of handwritten lead sheets produced without printed reference copies, with equivalent chord spellings and free layout, and compare against an independent human transcription. If word or line error rates move well above 11.90/29.68, or chord-note alignment fails, the model's generalisation is limited to the dataset's layout-matched, reference-copy setup.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the encoder-decoder architecture and the pretrained checkpoint that the experiments show is necessary for effective training."},{"cited_title":"Sheet music transformer: End-to-end optical music recogni- tion beyond monophonic transcription,","cited_arxiv_id":null,"evidence_quote":"Defines the convolutional encoder that compresses the staff image into a hidden representation."},{"cited_title":"Humdrum and Kern: Selective Feature En- coding BT - Beyond MIDI: The handbook of musi- cal codes,","cited_arxiv_id":null,"evidence_quote":"Provides the transformer decoder that autoregressively generates the token sequence from the image representation."},{"cited_title":"Filosax: A dataset of annotated jazz saxophone recordings,","cited_arxiv_id":null,"evidence_quote":"Proposes the chord-syntax notation that the paper restricts into its canonical root:shorthand(extensions)/bass form."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the **kern text format on which the tokenisers, reconstruction, and evaluation metrics operate."},{"cited_title":"maj7” or “∆7","cited_arxiv_id":null,"evidence_quote":"Presents the aligned melody-and-lyrics approach whose vertical-alignment assumption fails on handwritten lead sheets, motivating the chosen architecture."},{"cited_title":"a” corresponds to very dif- ferent graphical symbols in the melody spine (a note) and chord spine (part of the “maj","cited_arxiv_id":null,"evidence_quote":"Establishes the polyphonic piano OMR setting from which the pretraining benefits are drawn."},{"cited_title":"A holistic approach for aligned music and lyrics transcription,","cited_arxiv_id":null,"evidence_quote":"Provides the region-detection model used to obtain the staff bounding boxes that define the region-level task."}],"review_version":1}