REVIEW 2 major objections 5 minor 31 references
Legato 2 reads sheet music system by system, recovering both notes and embedded text, and sets a new mark for optical music recognition and musical question answering.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 02:11 UTC pith:HJCXTZ7S
load-bearing objection Real OMR advance: system-by-system neural reading plus text, with clean SOTA tables and a useful VLM-context result; synthetic-to-real gap is the main limit, not a collapse of the claim. the 2 major comments →
LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Legato 2 shows that system-by-system autoregressive recognition—segmenting a score, decoding each system into system-level ABC conditioned on left-truncated prior context, then converting back to standard ABC—produces lower OMR-NED than prior full-page neural models, a rule-based OMR system, and frontier vision-language models across multiple rendered and camera datasets, while also recovering embedded text for the first time in a neural OMR pipeline. Supplying those transcriptions as optional context further improves frontier models on MusiXQA and SSMR-Bench, establishing new state-of-the-art results for both recognition and downstream sheet-music understanding.
What carries the argument
The central mechanism is the system-level recognition loop: a detector extracts horizontal systems; a vision-language model autoregressively predicts each system’s system-level ABC given only the current system image and truncated previous systems; a rule-based converter merges the pieces into standard ABC; a byte-fallback tokenizer preserves titles and annotations instead of collapsing them to a placeholder. That loop is what enables longer documents, finer local detail, and text-aware output.
Load-bearing premise
The load-bearing premise is that a model trained almost entirely on synthetic rendered scores will transfer well enough to denser, noisier real camera and scan pages.
What would settle it
If, on a held-out set of real multi-system camera scores (especially dense multi-staff pages), Legato 2’s OMR-NED were no better than the previous full-page neural baseline, or if supplying its transcriptions failed to raise frontier VLM accuracy on MusiXQA’s hard split and SSMR-Bench, the central claim would be falsified.
If this is right
- Multi-page scores of arbitrary length can be transcribed without image concatenation or ad-hoc page-merge logic.
- Titles, composer names, and inline annotations become part of the same symbolic stream as the notes.
- Frontier vision-language models gain a practical external music-reading tool rather than having to invent OMR from pixels alone.
- Downstream analysis tools can start from cleaner machine-readable scores extracted from scans and photographs.
- Memory stays bounded at inference because only one system plus a fixed context window is needed at a time.
Where Pith is reading between the lines
- The same system-by-system pattern may transfer to other dense line-structured documents such as tablature, lead sheets, or historical mensural notation.
- If distribution shift is the main bottleneck, denser real-world training pairs would likely unlock more gain from larger models than further decoder scaling alone.
- A modular stack—specialist OMR feeding a general VLM—could become a template for other domains that already have symbolic intermediates (for example chemistry diagrams or circuit schematics).
- Until detector and recognizer are trained jointly, residual segmentation errors will remain a practical risk for high-stakes archival digitization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Legato 2, a modular OMR pipeline that segments sheet-music pages into systems with a fine-tuned YOLO detector, then autoregressively transcribes each system with a vision-language model conditioned on left-truncated prior system-level ABC, followed by rule-based conversion to standard ABC. It is the first large neural OMR system to process system-by-system (enabling multi-page scaling) and the first to emit embedded text (titles, composers, annotations) via a byte-fallback BPE tokenizer. Evaluations on PDMX-Synth, rendered/camera OpenScore String Quartets and Lieder, and IMSLP Piano report lower OMR-NED than Legato 1, Audiveris, and frontier VLMs; text CER is also lower than Audiveris/Gemini/PaddleOCR. Supplying the resulting transcriptions as optional context further raises G-Acc on MusiXQA and accuracy on SSMR-Bench for Gemini and GPT-5. Ablations isolate gains from system segmentation, byte fallback, and vocabulary size 4096.
Significance. If the reported gains hold under the stated evaluation protocol, the work advances practical OMR by replacing full-page recognition with a sequential, context-aware system-level model that scales to long scores without image concatenation and that finally recovers textual metadata inside a neural pipeline. The demonstration that imperfect symbolic transcriptions measurably improve frontier VLM performance on dense musical VQA (MusiXQA OMR split, SSMR-Bench) is a concrete, reusable contribution to multimodal music understanding. Strengths include multi-dataset external testing (rendered and camera), multi-page length-binned evaluation with confidence intervals, transparent validation-set hyperparameter selection, component ablations, and an explicit distribution-shift diagnosis. These elements make the pipeline a credible new baseline for both recognition and tool-augmented music reasoning.
major comments (2)
- [Table 1, §3.3.1, Appendix E.1] Table 1 (Camera OpenScore String Quartets / Lieder and IMSLP Piano rows) and §3.3.1 / Appendix E.1: OMR-NED is a rendered-measure set-edit distance. For the camera and IMSLP sets the symbolic ground truth is taken from separate OpenScore/IMSLP sources (or author annotation) rather than from the photographed pages themselves. Residual misalignment—cropping, missing systems, editorial variants, or incomplete annotation—would inflate absolute NED for every system and can differentially favor a system-level model that recovers partial content. The paper already notes density shift (Appendix F.2) and the synthetic-to-real gap (Limitations) but does not report image–GT registration quality or inter-annotator agreement on these sets. The large gains on the perfectly aligned rendered rows already support the SOTA claim; please either quantify registration fidelity for the camera/IMSLP rows or ex
- [§4.2, Figure 3] §4.2 and Figure 3: Multi-page evaluation is restricted to a direct comparison against Legato 1 (aspect-ratio bins, 500 sampled documents). While computational cost is a legitimate reason, the abstract and introduction claim robust long-document processing as a core advantage over prior methods. Without at least one additional baseline (e.g., page-independent Audiveris or a concatenated Legato-1 variant) on a subset of the bins, the multi-page SOTA claim remains only partially substantiated. A short additional experiment or a clearer scoping statement would close the gap.
minor comments (5)
- [Title page] Title and running header contain the split form “LEGA TO 2”; standardize to “Legato 2” throughout.
- [Table 2a, §4.3] Table 2a: composer CER remains high (~97 %) because page numbers are frequently misread as composer names. A short note on post-processing heuristics or a filtered “composer-only” metric would make the text-recognition claim easier to interpret.
- [Appendix A.3] Appendix A.3: the rule-based converter discards terminal systems on ill-formed output. Report the frequency of such discards on the test sets so readers can gauge how often the lossless-conversion assumption fails in practice.
- [Figures 1–2] Figure 1 and Figure 2 are dense; increasing font size of the ABC snippets or providing a simplified schematic would improve readability.
- [§2.4] §2.4 footnote 2: exclusion of lyrics is reasonable, but a one-sentence pointer to existing OCR pipelines that could be chained would strengthen the “text-aware” claim.
Circularity Check
No circularity: empirical OMR pipeline evaluated against held-out external benchmarks; Legato 1 self-citation is a baseline, not a load-bearing derivation.
full rationale
Legato 2 is a standard empirical systems paper. Its central claims (lower OMR-NED than Legato 1 / Audiveris / frontier VLMs on PDMX-Synth, OpenScore rendered+camera, and IMSLP Piano; higher G-Acc/accuracy when its transcriptions are supplied as context on MusiXQA and SSMR-Bench) are measured against held-out ground truth and external systems, not derived from quantities that already encode the target. Training is on PDMX-Synth; hyperparameters are chosen on a separate OpenScore validation split excluded from test; multi-page and text-recognition results use the same external evaluation protocol. System-level ABC is an intermediate representation with a rule-based converter to standard ABC, not a self-definitional identity that forces the reported NED. Architecture and tokenizer choices (Llama-3.2 vision backbone, byte-fallback BPE, YOLO segmentation) are design decisions ablated and reported, not uniqueness theorems imported from the authors. Citation of Legato 1 [27] is used as prior SOTA baseline and starting architecture—normal self-citation that does not force the new result by construction. No fitted parameter is renamed as a prediction; no ansatz is smuggled in as a theorem. Distribution-shift and GT-alignment concerns affect correctness risk, not circularity. Score 0 is therefore appropriate.
Axiom & Free-Parameter Ledger
free parameters (4)
- tokenizer vocabulary size =
4096
- inference beam size and repetition penalty =
beam=1, rep_penalty=1.1 (Legato 2)
- context left-truncation length =
1024
- YOLO fine-tune set size and mix =
1024 pages
axioms (4)
- ad hoc to paper System-level ABC can be losslessly converted to and from standard ABC via the stated rule-based procedure (modulo occasional ill-formed VLM outputs that are discarded).
- domain assumption Left-truncation of prior system context to 1024 tokens preserves enough local musical information for accurate next-system prediction.
- domain assumption Synthetic PDMX-Synth renderings plus limited real fine-tuning are a sufficient training distribution for real camera/scan evaluation.
- domain assumption OMR-NED (measure-level set edit distance on rendered symbols) is a faithful quality metric for the claimed SOTA comparisons.
invented entities (2)
-
system-level ABC
no independent evidence
-
text-aware ABC tokenizer with byte fallback
no independent evidence
read the original abstract
We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level segmentation with an autoregressive vision-LM to capture both local notation details and score structure. Across multiple datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.
Figures
Reference graph
Works this paper leans on
-
[1]
Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
AI@Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Technical report, Meta Platforms, Inc., September 2024. URL https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/
work page 2024
-
[2]
Do you actually know what classical music is? Does anyone?The At- lantic, April 2025
Matthew Aucoin. Do you actually know what classical music is? Does anyone?The At- lantic, April 2025. URL https://www.theatlantic.com/magazine/archive/2025/05/ aucoin-what-is-classical-music/682119/
work page 2025
-
[3]
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[4]
The challenge of optical music recognition.Computers and the Humanities, 35(2):95–121, May 2001
David Bainbridge and Tim Bell. The challenge of optical music recognition.Computers and the Humanities, 35(2):95–121, May 2001. ISSN 1572-8412. doi: 10.1023/A:1002485918032. URLhttps://doi.org/10.1023/A:1002485918032
-
[5]
Hervé Bitteur and Audiveris Contributors. Audiveris, 2025. URL https://github.com/ Audiveris/audiveris
work page 2025
-
[6]
Understanding optical music recognition.ACM Comput
Jorge Calvo-Zaragoza, Jan Haji ˇc Jr., and Alexander Pacha. Understanding optical music recognition.ACM Comput. Surv., 53(4), July 2020. ISSN 0360-0300. doi: 10.1145/3397499. URLhttps://doi.org/10.1145/3397499
-
[7]
MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models
Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Jiayu Qin, Ruiyi Zhang, and Changyou Chen. MusiXQA: Advancing visual music understanding in multimodal large language models, 2025. URLhttps://arxiv.org/abs/2506.23009
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[8]
PaddleOCR 3.0 technical report,
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. PaddleOCR 3.0 technical report,
-
[9]
URLhttps://arxiv.org/abs/2507.05595
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
Google DeepMind. Gemini 3.1 Pro model card. Model card, Google DeepMind, Febru- ary 2026. URL https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-1-Pro-Model-Card.pdf. Published February 2026
work page 2026
-
[11]
Google DeepMind. Gemini 3 flash model card. https://storage.googleapis.com/deepmind- media/Model-Cards/Gemini-3-Flash-Model-Card.pdf, 2026. Model card for the Gemini 3 Flash multimodal model
work page 2026
-
[12]
Mark Gotham, Maureen Redbond, Bruno Bower, and Peter Jonas. The “OpenScore String Quartet” corpus. InProceedings of the 10th International Conference on Digital Libraries for Musicology, pages 49–57, New York, NY , USA, 2023. Association for Computing Machinery. doi: 10.1145/3625135.3625155
-
[13]
Mark Robert Haigh Gotham and Peter Jonas. The OpenScore Lieder corpus. InMusic Encoding Conference Proceedings 2021, pages 131–136. Humanities Commons, 2022. doi: 10.17613/1my2-dm23
-
[14]
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, 2023. URL https:// github.com/ultralytics/ultralytics
work page 2023
-
[15]
Jongmin Jung, Dongmin Kim, Sihun Lee, Seola Cho, Hyungjoon So, Irmak Bukey, Chris Donahue, and Dasaem Jeong. U-MusT: A unified framework for cross-modal translation of score images, symbolic music, and performance audio.IEEE Transactions on Audio, Speech and Language Processing, pages 1–16, 2025. doi: 10.1109/TASLPRO.2025.3648794. 10
-
[16]
PDMX: A large-scale public domain musicxml dataset for symbolic music processing
Phillip Long, Zachary Novack, Taylor Berg-Kirkpatrick, and Julian McAuley. PDMX: A large-scale public domain musicxml dataset for symbolic music processing. InICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2025. doi: 10.1109/ICASSP49660.2025.10890217
-
[17]
Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation
Juan C. Martinez-Sevilla, Joan Cerveto-Serrano, Noelia Luna, Greg Chapman, Craig Sapp, David Rizo, and Jorge Calvo-Zaragoza. Sheet music benchmark: Standardized optical music recognition evaluation, 2025. URLhttps://arxiv.org/abs/2506.10488
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[18]
Practical end-to-end optical music recognition for pianoform music
Jiˇrí Mayer, Milan Straka, Jan Haji ˇc, and Pavel Pecina. Practical end-to-end optical music recognition for pianoform music. In Elisa H. Barney Smith, Marcus Liwicki, and Liangrui Peng, editors,Document Analysis and Recognition - ICDAR 2024, pages 55–73, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-70552-6
work page 2024
-
[19]
WildScore: Benchmarking MLLMs in-the-wild symbolic music reasoning
Gagan Mundada, Yash Vishe, Amit Namburi, Xin Xu, Zachary Novack, Julian McAuley, and Junda Wu. WildScore: Benchmarking MLLMs in-the-wild symbolic music reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, edi- tors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, pages 16847...
-
[20]
IMSLP petrucci music library.https://imslp.org, 2026
Project Petrucci LLC. IMSLP petrucci music library.https://imslp.org, 2026
work page 2026
-
[21]
MuPT: A generative symbolic music pretrained transformer
Xingwei Qu, yuelin bai, Yinghao MA, Ziya Zhou, Ka Man Lo, JIAHENG LIU, Ruibin Yuan, Lejun Min, Xueling Liu, Tianyu Zhang, Xeron Du, Shuyue Guo, Yiming Liang, Yizhi Li, Shangda Wu, Junting Zhou, Tianyu Zheng, Ziyang Ma, Fengze Han, Wei Xue, Gus Xia, Emmanouil Benetos, Xiang Yue, Chenghua Lin, Xu Tan, Wenhao Huang, Jie Fu, and Ge Zhang. MuPT: A generative s...
work page 2025
-
[22]
Ana Rebelo, Ichiro Fujinaga, Filipe Paszkiewicz, Andre R. S. Marcal, Carlos Guedes, and Jaime S. Cardoso. Optical music recognition: state-of-the-art and open issues.International Journal of Multimedia Information Retrieval, 1(3):173–190, Oct 2012. ISSN 2192-662X. doi: 10.1007/s13735-012-0004-6. URLhttps://doi.org/10.1007/s13735-012-0004-6
-
[23]
Sheet music transformer: End- to-end optical music recognition beyond monophonic transcription
Antonio Ríos-Vila, Jorge Calvo-Zaragoza, and Thierry Paquet. Sheet music transformer: End- to-end optical music recognition beyond monophonic transcription. In Elisa H. Barney Smith, Marcus Liwicki, and Liangrui Peng, editors,Document Analysis and Recognition - ICDAR 2024, pages 20–37, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-70552-6
work page 2024
-
[24]
End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music
Antonio Ríos-Vila, Jorge Calvo-Zaragoza, David Rizo, and Thierry Paquet. End-to-end full- page optical music recognition for pianoform sheet music, 2025. URL https://arxiv.org/ abs/2405.12105
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[25]
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirchmeyer, Ale...
-
[26]
URLhttps://arxiv.org/abs/2601.03267
work page internal anchor Pith review Pith/arXiv arXiv
-
[27]
Wong, Jizhe Zhou, and Yu Cheng
Zhilin Wang, Zhe Yang, Yun Luo, Yafu Li, Xiaoye Qu, Ziqian Qiao, Haoran Zhang, Runzhe Zhan, Derek F. Wong, Jizhe Zhou, and Yu Cheng. Towards an AI musician: Synthesizing sheet music problems for musical reasoning, 2025. URLhttps://arxiv.org/abs/2509.04059
-
[28]
Guang Yang, Muru Zhang, Lin Qiu, Yanming Wan, and Noah A. Smith. Toward a more complete omr solution. InProceedings of the 25th International Society for Music Information Retrieval Conference, pages 930–937. ISMIR, November 2024. doi: 10.5281/zenodo.14877483. URLhttps://doi.org/10.5281/zenodo.14877483
-
[29]
Guang Yang, Victoria Ebert, Nazif Tamer, Brian Siyuan Zheng, Luiza Pozzobon, and Noah A. Smith. Legato: Large-scale end-to-end generalizable approach to typeset OMR, 2025. URL https://arxiv.org/abs/2506.19065. A Pipeline Details A.1 Context Management Mechanism Our VLM is trained to model the probability P(a i |a <i, ui), where ai represents the system-le...
-
[30]
Analyze the Time Signature: The time signature at the beginning of the staff is 2/2…2. Analyze the Note Values: ... First group: Quarter note (1) + two eighth notes (0.5 + 0.5 = 1) + quarter note (1) = 3 qne... Fifth group:Quarter note (1) + two eighth notes (1) + quarter note (1) + quarter note (1) = 4 qne3. Evaluate the Choices:Choice A: … Choice B: … C...
-
[31]
Understand the Time Signature:... The time signature is 2/2 … each measure must contain exactly 4 quarter notes(or 8 eighth notes) in total duration.2. Analyze the Note Durations:... Group 1:4 eighth notes = 2 beats ... Group 4:1 quarter note = 1 beat. Group 5:2 slurred eighth notes = 1 beat … Group 8:4 eighth notes = 2 beats3. Group the Notes into Measur...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.