{"id":"f7066cff-960d-4ce4-bc3f-392f2652729b","arxiv_id":"2608.10978","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OSSQ-OMR is the first multi-part OMR dataset, pairing 116 aligned string quartet scores with three symbolic encodings, and its benchmark shows encoding choice matters more than architecture.","lead":"Researchers created the first dataset for teaching computers to read multi-part string quartet sheet music, pairing 116 digital scores with their original scanned pages. They also ran a benchmark showing encoding format matters more than model choice, with error rates under 6% on scanned pages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated manual visual alignment (§2.2) is the load-bearing assumption: without independent verification, every paired image-transcription ground truth and all baseline OMR-NED numbers are contingent on the authors' 100+ hours of subjective edits.","rationale":"I agree with the reader's weakest assumption: the manual alignment in §2.2 is the single condition on which the whole dataset and benchmark depend. I considered the two other weaknesses named by the reader (the large filtered pool in §3.3 and the cross-architecture hyperparameter confounds in §4.1). These are real but they affect only the breadth and interpretation of the benchmark conclusions, not the validity of the dataset artifact itself. An incorrect alignment, by contrast, invalidates every paired image-transcription sample and therefore every reported OMR-NED number. The paper does include useful independent infrastructure: a Git history that records every edit, round-trip verification for the format conversions in §2.4, and four mutually exclusive test splits. These supports reduce but do not remove the need for an external check of the alignment step. The proposed audit is deliberately feasible: 30 systems is small enough to be done by one annotator in days and large enough to cover the high-edit files that dominate the risk. If the audit passes, the conditional concern is resolved and the verdict could move to ACCEPT; if it fails, the benchmark conclusions would need to be recomputed on a cleanly aligned subset. Since neither outcome is established here, the reader's CONDITIONAL verdict is unchanged.","tokens_in":10967,"tokens_out":7128,"duration_ms":68208,"concrete_test":"Independent alignment audit: sample 30 systems stratified across the 93 full-score scans, deliberately including all 21 files with pitch edits and a mix of layout-only and notation-only files. Have a second annotator re-run the §2.2 alignment from the original OpenScore encoding to the IMSLP scan, blinded to the authors' Git history, and compute OMR-NED (Eq. 1) between the authors' aligned encoding and the independent alignment, restricted to the note, clef, tie, and dot categories. Also have the annotator mark any residual rendering/scan mismatch after alignment. If the independent alignment differs by more than ~2% OMR-NED on these categories (or if any of the 30 systems shows an unmarked mismatch), the released ground truth and the 3.6%/5.9% baseline figures must be re-derived on a verified subset before the benchmark is treated as definitive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's central claim is that OSSQ-OMR supplies token-level-accurate image-transcription pairs for multi-part OMR and that the reported baselines (3.6% synthetic, 5.9% scanned) establish feasibility. That claim rests on the manual visual alignment described in §2.2. The paper provides no quantitative validation of this alignment: no inter-annotator agreement, no automated render-to-scan correspondence check, and no independent audit. The authors state that they 'iteratively edited, reviewed, and validated the results' over 100+ hours, but this is a process description, not evidence of correctness. Table 1 shows large intervention counts (7,183 pitch edits in 21 files, 1,948 slur edits, 786 clef edits, 984 system-break edits), so the prior for human error or overcorrection is nontrivial. Because the same corrected MuseScore files produce both the synthetic training images and the symbolic ground truth for the scanned images, any systematic mismatch between renderings and scans propagates into the training distribution and the evaluation reference. The released Git history makes edits auditable in principle, but auditability is not validation. The phrase 'token level' is never given an operational definition, so the claim cannot currently be reproduced or checked from the paper alone. This is an internal completeness gap, not a disagreement with community consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OSSQ-OMR, a dataset and benchmark for multi-part optical music recognition (OMR). The dataset provides 24,544 system images and 98,172 staff images drawn from 116 synthetic and 93 scanned string quartet scores, paired with symbolic transcriptions in three formats (LMXE, **kern, ABC). The images and transcriptions are derived from OpenScore String Quartet encodings that have been manually edited so that MuseScore renderings visually match the IMSLP scans. The benchmark evaluates two baseline architectures (Zeus and Sheet Music Transformer) across nine tokenization schemes and four score-level splits, reporting OMR-NED as low as 3.6% on synthetic and 5.9% on scanned inputs. The paper also includes an external baseline comparison with Legato.","tokens_in":11262,"tokens_out":5939,"duration_ms":50477,"significance":"If the manual alignment is trustworthy, OSSQ-OMR fills a clear gap in the OMR literature: it is the first dataset dedicated to multi-part score transcription, with a substantial number of images and multiple symbolic encodings. The release of the corrected MuseScore sources as auditable Git history is a concrete strength, and the external Legato comparison provides an independent anchor for evaluating the baselines. The benchmark protocol is clearly structured and the result that encoding choices substantially affect accuracy is useful for the community. However, the central validity of the paired image-transcription ground truth rests on an unvalidated manual alignment process, and the benchmark pool excludes a large, unquantified fraction of the released data due to height and length caps. These issues must be addressed before the dataset and benchmark can be fully relied upon.","major_comments":[{"comment":"The claim that the rendered MuseScore images match the IMSLP scans 'at the token level' is not supported by any quantitative validation. The paper reports 100+ hours of iterative editing but no inter-annotator agreement, no automated render-to-scan correspondence check, and no independent audit. Since Table 1 records 7,183 pitch edits and 1,948 slur edits across a subset of the files, the prior for human error or overcorrection is nontrivial. Because the corrected MuseScore files are the single source for both the synthetic training images and the symbolic ground truth for scanned images, any alignment error propagates into the training distribution and the evaluation reference. The released Git history makes the edits auditable but does not by itself establish correctness. Please add a validation study, for example a second annotator on a random sample or an automated image-to-symbol correspondence metric, and provide an operational definition of 'token-level' alignment.","section":"Section 2.2, Table 1"},{"comment":"The benchmark pool is the intersection of samples passing the height cap and all per-encoding length caps, but the paper does not report how many samples are excluded by each filter or analyze the coverage bias. Table 3 is internally inconsistent: for system-level, train 13,552±128 and valid 1,611±1 already exceed the reported 'Total (synth)' of 10,016, so the table's meaning is unclear. The release counts in Table 2 (e.g., 13,240 synthetic system images) are far larger than the benchmark pool appears to be, which may imply that a large fraction of long systems or verbose encodings is absent from the evaluation. Please report the exact filtering funnel (raw -> height cap -> each encoding cap -> intersection) and characterize the excluded samples, since the baselines only speak to the filtered subset.","section":"Section 3.3, Table 3"},{"comment":"The headline robustness result ('the LSTM-based baseline degrading on scanned inputs roughly 2.6 times less than the Transformer-based baseline') is confounded: Zeus and SMT differ simultaneously in architecture, learning rate (0.001 with cosine schedule vs 0.0001 without), and, for Zeus, modified hidden sizes. The paper acknowledges that the architectural source of the gap is not isolated, but the abstract and conclusions state the result as an architecture comparison. Please either provide a controlled comparison with matched training hyperparameters or soften the attribution to the models as configured.","section":"Section 4.1, Section 4.2, Table 4"},{"comment":"The abstract states that 'all images visually aligned to their transcriptions,' but Section 2.2 says the alignment process 'was exclusively applied to the 93 full-score scans.' If scanned images from the 17 part-books and 6 manuscripts are included in the release without the same alignment, the claim is overbroad; if they are not included, the dataset's scanned coverage is only 93 of the 116 scores. Please clarify the status of part-book and manuscript scans and align the abstract with the actual alignment scope.","section":"Abstract and Section 2.2, Section 2.5"}],"minor_comments":[{"comment":"The meaning of the 'Total (synth)' and 'Total (scan)' rows is unclear; please rewrite the table to distinguish per-split counts from overall pool sizes.","section":"Table 3"},{"comment":"The robustness comparison reports both relative degradation (39% vs 100% increase) and absolute increases (3.6 pp vs 6.7 pp); the '2.6×' factor should be defined exactly to avoid ambiguity.","section":"Section 4.2"},{"comment":"The time-signature injection for **kern and ABC makes time-signature accuracy incomparable across segmentation levels; this is stated but should be reflected in the conclusions when comparing encodings.","section":"Section 3.4"},{"comment":"Reference [28] points to MuseScore.com rather than the MuseScore software; please correct the citation.","section":"References"},{"comment":"The comparison of Legato's system-wise OMR-NED on OSSQ-OMR to its previously reported page-wise result is only indicative because the segmentation levels differ; please state this explicitly.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the unvalidated manual alignment in Section 2.2. If the authors can supply a credible validation study, the paper would be a strong contribution to the OMR community. I would also ask them to fix the inconsistencies in Table 3, as they currently obscure the benchmark pool size. The dataset and benchmark are valuable and the paper is within scope for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OSSQ-OMR is the first dataset built specifically for multi-part OMR, pairing 116 string quartet scores from OpenScore with their IMSLP scans. That alone fills a real gap: prior datasets stopped at monophonic or piano-form. The paper also ships a reproducible benchmark, three encodings (LMXE, **kern, ABC), a segmentation pipeline, and released code and data. The strongest result—3.6% OMR-NED on synthetic, 5.9% on scanned at part level with Zeus—is plausible, and the external Legato comparison is a nice sanity check.\n\nThe load-bearing assumption is the manual visual alignment in §2.2. The authors report 100+ hours of edits, release the git history, and categorize edits, but they provide no inter-annotator agreement, no automated render-to-scan verification, and no operational definition of \"token-level.\" The paper is honest about the process, and auditability is a partial mitigation, but it is not validation. If the alignment is off, the paired labels and all baselines inherit the error. This is a genuine internal gap, not a nitpick.\n\nSecond, the benchmark pool is the intersection of all encodings and excludes any system above 256px or over per-token caps; Table 3 shows roughly 25% of synthetic and scanned systems dropped. Disclosed, but it means the benchmark numbers describe a filtered subset, and the cap interacts with score texture.\n\nThird, the Zeus vs SMT robustness comparison is confounded: Zeus gets larger hidden sizes and a different learning-rate schedule. So the \"LSTM degrades 2.6x less\" claim is suggestive, not solid. Minor.\n\nOn the other hand, the LMXE results are believable as an encoding effect, and the BPE monotonic degradation is a clean finding.\n\nThis paper is for OMR and MIR researchers, plus cultural heritage digitization. It deserves a serious referee. Even with the alignment caveat, a dataset release with this scale, paired scans, and a usable benchmark is worth engaging on the merits. Send it to peer review and ask for alignment validation or a clear statement of uncertainty, and for the filtered-pool effect on conclusions.","headline":"A genuinely first multi-part OMR dataset with a usable benchmark, but the manual alignment is the load-bearing assumption and the paper needs independent validation or a clear caveat.","tokens_in":11763,"tokens_out":2364,"would_cite":true,"duration_ms":21955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces OSSQ-OMR, the first dataset for multi-part optical music recognition, pairing 24,544 system and 98,172 staff images from 116 string quartet scores with token-aligned transcriptions, and records baselines of 3.6%…","keywords":["optical music recognition","multi-part scores","string quartet","dataset","benchmark","OMR-NED","LMXE","visual alignment"],"falsifier":"Take a random sample of, say, 20 of the 93 aligned full-score scans, have an independent annotator transcribe each scan from scratch into a symbolic format, and compute token-level agreement (e.g., OMR-NED) between the independent transcription and the released ground truth; if the agreement is far above the claimed baseline errors, the ground truth—and all benchmark numbers—are compromised.","tokens_in":10788,"feed_emoji":"🎼","tokens_out":5827,"duration_ms":44870,"temperature":0.7,"pith_summary":"Optical music recognition (OMR) has worked well for single-line and piano scores, but multi-part scores—where several instruments play simultaneously—have been held back by the absence of a dataset to train and test on. This paper introduces OSSQ-OMR, a dataset of 24,544 system-level and 98,172 staff-level images from 116 string quartet scores, each paired with a scanned edition and with token-aligned transcriptions in three symbolic encodings. The authors argue that because the dataset is public and comes with a benchmark protocol, multi-part OMR becomes a measurable and tractable problem. Their baseline experiments show the task is feasible, with best errors of 3.6% on synthetic and 5.9% on scanned images, while revealing that encoding and segmentation choices affect accuracy more than the choice of architecture.","feed_headline":"Multi-part OMR benchmark hits 3.6% error","feed_subtitle":"The first multi-part OMR dataset pairs 116 string quartets with three encodings so the task can be trained and measured.","key_machinery":"The load-bearing mechanism is the visual-alignment pipeline: the authors re-edit each OpenScore MuseScore file so that its MuseScore rendering aligns with the corresponding IMSLP scan at the token level, making the rendering and the symbolic transcription two views of the same ground truth. Around this core sit the YOLOv8-based segmentation pipeline that produces system and staff images, and the conversion pipeline that derives LMXE, **kern, and ABC from each corrected file with round-trip verification that removes samples failing structural or major musical checks. LMXE, an extension of Linearized MusicXML, is introduced to give a sequence-friendly, part-nesting encoding for multi-part scores.","core_discovery":"The paper's central claim is that OSSQ-OMR is the first dataset dedicated to multi-part OMR and that it makes multi-part score transcription a feasible, benchmarkable task. The dataset pairs 93 scanned full scores from IMSLP with corrected MuseScore encodings that were manually re-edited so that the rendered image matches the scan at the token level; from these, synthetic and scanned images are segmented into systems and staves, and transcriptions are released in LMXE, **kern, and ABC. Baseline experiments with two representative models and nine tokenization schemes report OMR-NED as low as 3.6% on synthetic and 5.9% on scanned inputs, and show that part-level inputs outperform system-level inputs in every configuration, that LMXE beats the other encodings, and that the LSTM-based model degrades on scanned inputs roughly 2.6 times less than the Transformer-based model.","pith_inferences":["If the alignment is as reliable as claimed, the same re-edit-and-render workflow could extend the corpus to other OpenScore genre sets, turning the entire OpenScore library into a multi-part OMR resource.","The consistent part-level advantage suggests current sequence models are not exploiting cross-part context; a model that jointly decodes the four staves of a system, rather than four independent sequences, is a natural next test.","The monotonic degradation of ABC-BPE with vocabulary size hints that for structured symbolic outputs, character-level or token-level supervision beats subword compression; this is a testable hypothesis that could generalize to other symbolic-sequence tasks.","A blind re-annotation of a random sample of the 93 aligned scores, performed without access to the authors' edits, would independently quantify the alignment's accuracy and is the direct way to check the dataset's ground truth."],"forward_implications":["With a shared public benchmark, multi-part OMR results across models and encodings become directly comparable, so progress on this task can be measured rather than asserted.","The reported results imply that representation choice—encoding and segmentation level—matters more than the encoder–decoder architecture: practitioners should prioritize LMXE and part-level inputs over model swaps.","The scan-vs-synthetic gap (3.6% to 5.9% for the best configuration) quantifies the remaining domain-shift problem, giving future work a concrete target.","Because the corrected MuseScore sources are released as an auditable Git history, the dataset supports not only training but also studies of how notation and layout edits affect OMR output."],"supporting_citations":[{"why":"Supplies the source corpus: all 116 digital string quartet scores that the dataset is built from.","marker":"[25]"},{"why":"Provides the scanned editions from IMSLP that form the scanned-image half of the image-transcription pairs.","marker":"[26]"},{"why":"Defines the Zeus LSTM-based baseline model used for the benchmark experiments.","marker":"[30]"},{"why":"Defines the Sheet Music Transformer baseline model used for comparison and originally designed for **kern.","marker":"[11]"},{"why":"Supplies the YOLOv8 object detection architecture used for system and staff segmentation.","marker":"[27]"},{"why":"Provides the pre-trained staff-height regression model used as-is in the segmentation pipeline.","marker":"[29]"},{"why":"Defines the OMR-NED evaluation metric used for all baseline comparisons and round-trip verification.","marker":"[34]"},{"why":"Supplies the Legato external ABC-based baseline used to compare against a pretrained model.","marker":"[33]"},{"why":"Renders the corrected MuseScore files into the synthetic images used for training and evaluation.","marker":"[28]"}],"fun_headline_variants":["First multi-part OMR dataset: 3.6% error on synthetic","Multi-part OMR: first dataset, 3.6% NED baseline","New OMR benchmark for string quartets: 3.6% error","24k system images, 116 quartets: OMR's first multi-part test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's ground truth rides on the authors' manual visual alignment being correct at the token level: the 100+ hours of edits to 93 scores were not checked by an independent annotator or automated audit, so an alignment error would propagate into every paired image, transcription, and benchmark number.","fun_headline_variants_meta":{"raw":{"variants":["First multi-part OMR dataset: 3.6% error on synthetic","Multi-part OMR: first dataset, 3.6% NED baseline","New OMR benchmark for string quartets: 3.6% error","24k system images, 116 quartets: OMR's first multi-part test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0009,"raw_usage":{"total_tokens":3897,"prompt_tokens":988,"completion_tokens":2909,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2823}},"tokens_in":604,"tokens_out":2909,"duration_ms":24720,"temperature":1.0,"reasoning_tokens":2823,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:56:56.797295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 20 of the 93 aligned full-score scans, have an independent annotator transcribe each scan from scratch into a symbolic format, and compute token-level agreement (e.g., OMR-NED) between the independent transcription and the released ground truth; if the agreement is far above the claimed baseline errors, the ground truth—and all benchmark numbers—are compromised.","supporting_citations":[{"cited_title":"End-to-end full-page optical music recognition for pianoform sheet music,","cited_arxiv_id":null,"evidence_quote":"Supplies the source corpus: all 116 digital string quartet scores that the dataset is built from."},{"cited_title":"Optical music recognition by long short-term mem- ory networks,","cited_arxiv_id":null,"evidence_quote":"Provides the scanned editions from IMSLP that form the scanned-image half of the image-transcription pairs."},{"cited_title":"Camera-primus: Neu- ral end-to-end optical music recognition on realistic monophonic scores","cited_arxiv_id":null,"evidence_quote":"Defines the Zeus LSTM-based baseline model used for the benchmark experiments."},{"cited_title":"All dataset construction, experimental design, training, evalu- ation, and analysis were carried out by the authors, who verified all reported results","cited_arxiv_id":null,"evidence_quote":"Defines the Sheet Music Transformer baseline model used for comparison and originally designed for **kern."},{"cited_title":"End-to-end optical music recognition using neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv8 object detection architecture used for system and staff segmentation."},{"cited_title":"An empirical evaluation of end-to-end polyphonic optical music recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained staff-height regression model used as-is in the segmentation pipeline."},{"cited_title":"Digi- tization of choirbooks in guatemala,","cited_arxiv_id":null,"evidence_quote":"Defines the OMR-NED evaluation metric used for all baseline comparisons and round-trip verification."},{"cited_title":"Encoded spanish music heritage through verovio: The online platforms fondo de música tradicional imf–csic and books of hispanic polyphony,","cited_arxiv_id":null,"evidence_quote":"Supplies the Legato external ABC-based baseline used to compare against a pretrained model."},{"cited_title":"A sequence-to-sequence frame- work based on transformer with masked language model for optical music recognition,","cited_arxiv_id":null,"evidence_quote":"Renders the corrected MuseScore files into the synthetic images used for training and evaluation."}],"review_version":1}