{"id":"3f479791-76a9-4f3a-88c8-84ade60df85f","arxiv_id":"2607.05769","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"System-by-system autoregressive OMR with text-aware ABC transcription outperforms prior neural and rule-based systems and boosts VLM sheet-music QA.","lead":"Legato 2 is a system-by-system optical music recognition pipeline that segments sheet music, transcribes each system with a vision-language model conditioned on prior context, and recovers embedded text. It sets new empirical SOTA on OMR benchmarks and improves frontier VLMs on music understanding by supplying symbolic context.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"SOTA OMR claim rests on imperfect ground-truth alignment for real camera/scan pages, so reported OMR-NED gains may partly reflect evaluation artifacts rather than pure recognition quality.","rationale":"The reader correctly flags synthetic-to-real distribution shift as the main generalization risk and assigns CONDITIONAL pending code/data. That concern is real (Appendix F.2 density plots; Limitations). The more load-bearing threat to the strongest claim, however, is not merely that the model was trained on synthetic data, but that the evaluation metric on the real camera/scan sets may itself be noisy because OMR-NED assumes clean image–GT correspondence. Without a registration audit, the large gains on Camera OpenScore String Quartets (58.2 → 31.6) and IMSLP Piano (44.3 → 34.2) cannot be fully trusted as pure recognition improvements. The multi-page and understanding results are less affected (they use rendered or downstream VLM accuracy), so the paper’s contribution remains solid; the verdict stays CONDITIONAL, but the condition should include a GT-alignment check on the real sets, not only code release and more real training data. I therefore agree with the reader’s overall stance while shifting the weakest link from training distribution alone to evaluation validity on the real test distributions.","tokens_in":26564,"tokens_out":714,"duration_ms":8505,"concrete_test":"On Camera OpenScore String Quartets (252 pages), manually audit a stratified sample of 30 pages: for each page, render the GT ABC to a clean image, overlay/register it to the camera scan, and count systems/measures that cannot be put into 1–1 correspondence (missing, extra, or editorially different). Recompute Table 1 OMR-NED after discarding or re-aligning those pages. If Legato 2’s relative gain over Legato 1 shrinks by >30% of the reported gap (or absolute NED rises >5 points after cleaning), the camera/IMSLP SOTA claim is overstated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central SOTA claim (Table 1: Legato 2 lower OMR-NED than Legato 1 / Audiveris / Gemini across PDMX-Synth, OpenScore rendered+camera, IMSLP Piano) treats OMR-NED as a faithful measure of recognition quality. OMR-NED is a rendered-measure set-edit distance (Appendix E.1). For PDMX-Synth and rendered OpenScore this is well-defined. For Camera OpenScore and IMSLP Piano Scores, however, the paper uses photocopies/scans whose ground-truth ABC/MusicXML comes from separate OpenScore/IMSLP sources (or author annotation). Any residual misalignment between the photographed page and the symbolic GT—cropping, missing systems, editorial differences, layout variants, or incomplete author annotation—will inflate absolute NED for every system and can differentially favor a system-level model that recovers partial systems over a full-page model. The paper already flags density shift (Appendix F.2) and synthetic-to-real gap (Limitations), but does not report inter-annotator or image–GT registration quality for the camera/IMSLP sets. If those sets contain non-negligible alignment noise, the headline “consistently outperforms prior SOTA” is only partially supported by the numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces Legato 2, a modular OMR pipeline that segments sheet-music pages into systems with a fine-tuned YOLO detector, then autoregressively transcribes each system with a vision-language model conditioned on left-truncated prior system-level ABC, followed by rule-based conversion to standard ABC. It is the first large neural OMR system to process system-by-system (enabling multi-page scaling) and the first to emit embedded text (titles, composers, annotations) via a byte-fallback BPE tokenizer. Evaluations on PDMX-Synth, rendered/camera OpenScore String Quartets and Lieder, and IMSLP Piano report lower OMR-NED than Legato 1, Audiveris, and frontier VLMs; text CER is also lower than Audiveris/Gemini/PaddleOCR. Supplying the resulting transcriptions as optional context further raises G-Acc on MusiXQA and accuracy on SSMR-Bench for Gemini and GPT-5. Ablations isolate gains from system segmentation, byte fallback, and vocabulary size 4096.","tokens_in":26971,"tokens_out":1214,"duration_ms":22506,"significance":"If the reported gains hold under the stated evaluation protocol, the work advances practical OMR by replacing full-page recognition with a sequential, context-aware system-level model that scales to long scores without image concatenation and that finally recovers textual metadata inside a neural pipeline. The demonstration that imperfect symbolic transcriptions measurably improve frontier VLM performance on dense musical VQA (MusiXQA OMR split, SSMR-Bench) is a concrete, reusable contribution to multimodal music understanding. Strengths include multi-dataset external testing (rendered and camera), multi-page length-binned evaluation with confidence intervals, transparent validation-set hyperparameter selection, component ablations, and an explicit distribution-shift diagnosis. These elements make the pipeline a credible new baseline for both recognition and tool-augmented music reasoning.","major_comments":[{"comment":"Table 1 (Camera OpenScore String Quartets / Lieder and IMSLP Piano rows) and §3.3.1 / Appendix E.1: OMR-NED is a rendered-measure set-edit distance. For the camera and IMSLP sets the symbolic ground truth is taken from separate OpenScore/IMSLP sources (or author annotation) rather than from the photographed pages themselves. Residual misalignment—cropping, missing systems, editorial variants, or incomplete annotation—would inflate absolute NED for every system and can differentially favor a system-level model that recovers partial content. The paper already notes density shift (Appendix F.2) and the synthetic-to-real gap (Limitations) but does not report image–GT registration quality or inter-annotator agreement on these sets. The large gains on the perfectly aligned rendered rows already support the SOTA claim; please either quantify registration fidelity for the camera/IMSLP rows or ex","section":"Table 1, §3.3.1, Appendix E.1"},{"comment":"§4.2 and Figure 3: Multi-page evaluation is restricted to a direct comparison against Legato 1 (aspect-ratio bins, 500 sampled documents). While computational cost is a legitimate reason, the abstract and introduction claim robust long-document processing as a core advantage over prior methods. Without at least one additional baseline (e.g., page-independent Audiveris or a concatenated Legato-1 variant) on a subset of the bins, the multi-page SOTA claim remains only partially substantiated. A short additional experiment or a clearer scoping statement would close the gap.","section":"§4.2, Figure 3"}],"minor_comments":[{"comment":"Title and running header contain the split form “LEGA TO 2”; standardize to “Legato 2” throughout.","section":"Title page"},{"comment":"Table 2a: composer CER remains high (~97 %) because page numbers are frequently misread as composer names. A short note on post-processing heuristics or a filtered “composer-only” metric would make the text-recognition claim easier to interpret.","section":"Table 2a, §4.3"},{"comment":"Appendix A.3: the rule-based converter discards terminal systems on ill-formed output. Report the frequency of such discards on the test sets so readers can gauge how often the lossless-conversion assumption fails in practice.","section":"Appendix A.3"},{"comment":"Figure 1 and Figure 2 are dense; increasing font size of the ABC snippets or providing a simplified schematic would improve readability.","section":"Figures 1–2"},{"comment":"§2.4 footnote 2: exclusion of lyrics is reasonable, but a one-sentence pointer to existing OCR pipelines that could be chained would strengthen the “text-aware” claim.","section":"§2.4"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a solid, well-executed incremental advance over Legato 1 rather than a conceptual leap; the system-level design and text-aware tokenizer are genuine firsts for large neural OMR and the VLM-context results are useful. The camera-set alignment concern is real but does not overturn the rendered-set gains, so minor revision is appropriate. Scope fits a CV / multimedia journal that already publishes OMR and document-understanding work."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a solid, incremental methods paper that actually moves OMR. Two firsts land—large neural system-by-system sequential recognition, and neural recovery of embedded text—and the multi-page and VLM-context results are the parts I would use.\n\nWhat is new is not the base VLM (Legato 1 / Llama-3.2 vision) but the pipeline: YOLO systems, autoregressive conditioning on prior system-level ABC, a rule-based merge back to standard ABC, and a byte-fallback text-aware tokenizer. Ablations isolate segmentation as the big win; byte fallback adds text without wrecking music NED; vocab 4096 is chosen on a held-out real-ish validation set. Tables are consistent across PDMX-Synth, rendered and camera OpenScore, and IMSLP piano. Multi-page bins with CIs show slower degradation than full-page Legato 1. Text CER beats Audiveris/Gemini/PaddleOCR. Feeding their ABC to Gemini/GPT-5 lifts MusiXQA and SSMR-Bench. That is honest engineering with external benchmarks, not circular self-proof.\n\nSoft spots, in proportion. Training is almost all synthetic PDMX-Synth; they say so, plot the density shift, and note limited gains from deeper/wider decoders. That is the real bottleneck, not a hidden flaw. The stress-test on camera/IMSLP GT alignment is fair: OMR-NED is rendered-measure edit distance, and photocopy-to-OpenScore registration is not audited. Absolute NED on camera sets could be noisy, and a system-level model might look better under partial misalignment. But the same ranking holds on clean rendered sets, so the SOTA claim is not only an artifact. Code/data are promised, not yet out—standard for this stage. Architecture scaling is thin; data is the lever.\n\nWho it is for: people building digitization pipelines, music-AI tooling, or multimodal music QA. Not a general CV theory paper. Math is standard; citations are appropriate (Legato 1 as baseline, Audiveris, OpenScore, YOLO). Serious thinking, clear limitations section.\n\nI would send it to peer review. Expect referees to push on real-data training and GT registration for camera sets, not to desk-reject. Worth engaging if you care about OMR or music multimodality; cite the system-level + text + VLM-context results once code is public.","headline":"Real OMR advance: system-by-system neural reading plus text, with clean SOTA tables and a useful VLM-context result; synthetic-to-real gap is the main limit, not a collapse of the claim.","tokens_in":27526,"tokens_out":631,"would_cite":true,"duration_ms":18624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Legato 2 reads sheet music system by system, recovering both notes and embedded text, and sets a new mark for optical music recognition and musical question answering.","keywords":["optical music recognition","sheet music","vision-language models","system-level ABC","symbolic music","multimodal understanding","autoregressive transcription","embedded text recognition"],"falsifier":"If, on a held-out set of real multi-system camera scores (especially dense multi-staff pages), Legato 2’s OMR-NED were no better than the previous full-page neural baseline, or if supplying its transcriptions failed to raise frontier VLM accuracy on MusiXQA’s hard split and SSMR-Bench, the central claim would be falsified.","tokens_in":27504,"feed_emoji":"🎼","tokens_out":1029,"duration_ms":27538,"temperature":0.7,"pith_summary":"This paper argues that machines read sheet music better when they follow the score the way musicians do: one horizontal system at a time, carrying forward only the recent musical context. Legato 2 first cuts a page into systems with a detector, then uses a vision-language model to turn each system into symbolic ABC notation while conditioning on prior systems, and finally stitches those pieces into a standard score. Unlike earlier neural OMR systems, it keeps titles, composers, and inline annotations inside the same stream instead of discarding them. Across rendered and camera pages the pipeline lowers recognition error relative to full-page neural models and a rule-based baseline, and the same transcriptions, when given as optional context, raise frontier vision-language models on hard music-understanding benchmarks. A sympathetic reader cares because sheet music is still the authoritative medium of creation and study for large musical traditions; a reliable bridge from image to editable notation plus usable symbolic context is what lets general models actually reason about dense scores rather than guess from pixels.","feed_headline":"Sheet music read system by system beats full-page OMR","feed_subtitle":"Symbolic ABC with titles and annotations also lifts frontier models on musical questions.","key_machinery":"The central mechanism is the system-level recognition loop: a detector extracts horizontal systems; a vision-language model autoregressively predicts each system’s system-level ABC given only the current system image and truncated previous systems; a rule-based converter merges the pieces into standard ABC; a byte-fallback tokenizer preserves titles and annotations instead of collapsing them to a placeholder. That loop is what enables longer documents, finer local detail, and text-aware output.","core_discovery":"Legato 2 shows that system-by-system autoregressive recognition—segmenting a score, decoding each system into system-level ABC conditioned on left-truncated prior context, then converting back to standard ABC—produces lower OMR-NED than prior full-page neural models, a rule-based OMR system, and frontier vision-language models across multiple rendered and camera datasets, while also recovering embedded text for the first time in a neural OMR pipeline. Supplying those transcriptions as optional context further improves frontier models on MusiXQA and SSMR-Bench, establishing new state-of-the-art results for both recognition and downstream sheet-music understanding.","pith_inferences":["The same system-by-system pattern may transfer to other dense line-structured documents such as tablature, lead sheets, or historical mensural notation.","If distribution shift is the main bottleneck, denser real-world training pairs would likely unlock more gain from larger models than further decoder scaling alone.","A modular stack—specialist OMR feeding a general VLM—could become a template for other domains that already have symbolic intermediates (for example chemistry diagrams or circuit schematics).","Until detector and recognizer are trained jointly, residual segmentation errors will remain a practical risk for high-stakes archival digitization."],"forward_implications":["Multi-page scores of arbitrary length can be transcribed without image concatenation or ad-hoc page-merge logic.","Titles, composer names, and inline annotations become part of the same symbolic stream as the notes.","Frontier vision-language models gain a practical external music-reading tool rather than having to invent OMR from pixels alone.","Downstream analysis tools can start from cleaner machine-readable scores extracted from scans and photographs.","Memory stays bounded at inference because only one system plus a fixed context window is needed at a time."],"fun_headline_variants":["Legato 2 reads scores system by system, beating full-page OMR","First neural OMR with titles and annotations via sequential systems","System-level autoregressive OMR tops prior models across datasets","Symbolic ABC from systems lifts frontier models on music QA","Legato 2 sets new OMR and sheet-music understanding SOTA"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The load-bearing premise is that a model trained almost entirely on synthetic rendered scores will transfer well enough to denser, noisier real camera and scan pages.","fun_headline_variants_meta":{"raw":{"variants":["Legato 2 reads scores system by system, beating full-page OMR","First neural OMR with titles and annotations via sequential systems","System-level autoregressive OMR tops prior models across datasets","Symbolic ABC from systems lifts frontier models on music QA","Legato 2 sets new OMR and sheet-music understanding SOTA"]},"model":"grok-4.5","effort":"low","cost_usd":0.004158,"raw_usage":{"total_tokens":1245,"prompt_tokens":780,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":41580000,"prompt_tokens_details":{"text_tokens":780,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":392,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":780,"tokens_out":73,"duration_ms":4157,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T02:11:24.872257+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If, on a held-out set of real multi-system camera scores (especially dense multi-staff pages), Legato 2’s OMR-NED were no better than the previous full-page neural baseline, or if supplying its transcriptions failed to raise frontier VLM accuracy on MusiXQA’s hard split and SSMR-Bench, the central claim would be falsified.","supporting_citations":[],"review_version":1}