{"id":"6dd50316-6621-4460-981e-654a8a7c3004","arxiv_id":"2506.23009","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A synthetic music sheet QA dataset and a LoRA-fine-tuned Phi-3 model show large accuracy gains on OMR and chord tasks, but only within the synthetic distribution.","lead":"This paper introduces MusiXQA, a synthetic music sheet dataset with structured annotations, and Phi-3-MusiX, a fine-tuned multimodal model. It reports that GPT-4o and other MLLMs struggle on music sheet VQA, while the fine-tuned model scores far higher within the synthetic benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains are demonstrated only on synthetic MusiXTeX sheets from the same generation pipeline as training; no real-sheet evaluation supports the claim of music-sheet understanding, so external validity is the load-bearing untested assumption.","rationale":"The reader's weakest assumption is also the most load-bearing: the benchmark's only evidence for 'understanding' comes from synthetic sheets. I agree with that identification. A second-order concern is internal consistency: Section 1 says 9,600 sheets and 130k QA pairs, while Section 3.1 says 96k sheets and 1.34M QA pairs. These are not cosmetic; they change the dataset's scale by 10x and make reproducibility checks harder. However, resolving that discrepancy would not salvage the general claim if real-sheet performance is poor. The GPT-4o-as-judge metric and the 'oracle' caption in Table 1 are also worth fixing, but they are secondary because the magnitude of the gap is large and the within-distribution comparison is plausible. The paper deserves credit for a controlled generation scheme, two output-format ablations, and clear task definitions, but those strengthen the synthetic benchmark, not the transfer claim. Since the reader already returned CONDITIONAL and my concern matches theirs, the verdict should remain UNCHANGED: the paper should be accepted only after either real-sheet evaluation is added or the claims are explicitly limited to synthetic notation and data are released.","tokens_in":22698,"tokens_out":6801,"duration_ms":71210,"concrete_test":"Evaluate the released Phi-3-MusiX checkpoint on OMR bar-level pitch/duration extraction and chord estimation using real-sheet ground-truth data, e.g., PrIMuS/Camera-PrIMuS, DeepScores, or 100 manually annotated pages of real engraved music. Compare G-Acc against the GPT-4o+RAG baseline under identical prompts. If Phi-3-MusiX's relative advantage over GPT-4o collapses from about 8x to near parity, or its absolute G-Acc falls below roughly 30%, the synthetic-to-real transfer assumption fails and the headline claim must be restricted to synthetic sheets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that Phi-3-MusiX learns symbolic music reading (Abstract; Section 5) is supported only by Table 1, whose test images are rendered by the same MusiXTeX pipeline and the same chord-based heuristics used to generate the training data (Sections 3.2-3.4). Section 6 concedes that the music is generated via chord-based heuristics rather than real compositions. The 68.4 G-Acc on OMR and 84.9 on chord estimation, and the 8x/6x relative gains over GPT-4o baselines, are therefore in-distribution measurements on synthetic notation. No evaluation on real engraved, scanned, or handwritten sheet music is reported, so the general phrase 'music sheet understanding' is not established. If synthetic-to-real transfer is low, the paper's contribution reduces to a synthetic benchmark rather than visual music understanding. This concern is compounded by the scale contradiction between Section 1 (9,600 sheets, 130k QA pairs) and Section 3.1 (96k sheets, 1.34M QA pairs), which makes the dataset impossible to verify from the text, but the decisive missing evidence is external validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MusiXQA, a large synthetic dataset of music sheet images rendered with MusiXTeX, together with template-based QA pairs covering OCR, layout understanding, OMR, and chord estimation. The authors benchmark several open and proprietary MLLMs, report that GPT-4o and related baselines perform poorly on the OMR and chord tasks, and propose Phi-3-MusiX, a LoRA fine-tuned Phi-3-Vision model. They report large improvements over the GPT-4o baselines, especially when notes are represented in the proposed kern+ format rather than JSON, and they attribute the gain partly to the compactness of the output representation. The paper also includes a limitation section acknowledging that the music is generated by chord-based heuristics rather than real compositions.","tokens_in":22909,"tokens_out":2522,"duration_ms":28960,"significance":"If the claims hold, the paper would provide a useful synthetic resource for studying music sheet understanding in MLLMs, as well as an interesting empirical finding about output-token format in structured prediction. Several strengths deserve credit: the annotation pipeline derives ground truth directly from the MusiXTeX source, avoiding post-processing errors; the generation framework supports controlled sampling across clefs, keys, time signatures, tempo, and layout variations; the comparison between JSON and kern+ is a concrete, reproducible experimental design; and the paper honestly discloses the synthetic nature of the data in Section 6. The central limitation is that all reported gains are measured in-distribution on the same synthetic pipeline used for training, so the external validity of the headline 'music sheet understanding' claim is untested. The dataset-statistics contradictions also prevent verification of the benchmark scale.","major_comments":[{"comment":"The central claim that Phi-3-MusiX learns 'music sheet understanding' is supported only by evaluations on synthetic MusiXTeX sheets generated with the same chord-based heuristics used for training. Section 6 concedes that the music is generated via heuristics rather than real compositions, and no evaluation is reported on existing real-sheet benchmarks such as PrIMuS, DeepScores, Camera-PrIMuS, or MUSCIMA++, nor on a held-out set of real engraved scores. As it stands, the 8x/6x G-Acc gains in Table 1 may reflect in-distribution overfitting to the synthetic rendering and annotation style rather than transferable music-reading ability. I ask the authors to either (a) add a real-sheet evaluation or a careful synthetic-to-real transfer experiment, or (b) substantially soften the 'music sheet understanding' claim to 'synthetic music sheet understanding' throughout the abstract and conclusion.","section":"§6, §3.3, §5"},{"comment":"The dataset size is reported inconsistently: Section 1 states 9,600 sheets and 'over 130,000' QA pairs, while Section 3.1 reports 96k sheets, 1.3M bars, 11.7M notes, and 670k + 337k + 288k + 47k = 1.342M QA pairs. This is a factor-of-ten discrepancy in sheets and roughly a factor-of-ten discrepancy in QA pairs. Because the paper's contribution is explicitly the dataset and because the training split of the model is drawn from this dataset, the reader cannot determine which numbers are correct. Please correct the abstract/introduction or the statistics section, and specify whether the 90/10 split in Section 4.1 applies to sheets, QA pairs, or both.","section":"§1 vs. §3.1"},{"comment":"The Table 1 caption refers to 'GPT-4o + RAG + OMR' as using 'oracle OMR results,' but Section 4.1 states that the OMR output comes from the Oemer model, and Section 4.5 explicitly notes that Oemer is lossy and omits accidentals. Calling a lossy model's output 'oracle' is misleading and inflates the apparent strength of the baseline that Phi-3-MusiX is compared against. If an oracle OMR baseline is intended, it should use ground-truth annotations from the MusiXTeX source; otherwise the caption and the main-text description should consistently say 'OMR-augmented' rather than 'oracle.'","section":"Table 1 caption, §4.1, §4.5"},{"comment":"The fine-tuned model is trained and evaluated on the same MusiXQA distribution, so the reported improvements are closed-loop with respect to that distribution. The zero-shot GPT-4o baselines provide some independent grounding that the task is not trivially easy, but they do not establish that the learned representation generalizes beyond the specific synthetic annotation and rendering style. I recommend that the authors report at least one cross-dataset or human-annotated real-score condition, or explicitly describe the contribution as a synthetic benchmark with in-distribution gains, rather than as general visual music understanding.","section":"§4.4, §5"}],"minor_comments":[{"comment":"The title contains a typo: 'ADV ANCING' should be 'ADVANCING.'","section":"Title"},{"comment":"In Figure 3, the label 'Temp o' should be 'Tempo,' and the example for the chord label shows inconsistent quoting ('Bb chord' with straight and curly quotes) that should be normalized.","section":"Figure 3"},{"comment":"The description of the JSON representation says an underscore marks 'the start of a slur' and a dot indicates dotted rhythms, but Figure 4 uses an underscore after a duration and a dot after a note value; please clarify the exact token semantics in one consistent place.","section":"§4.2"},{"comment":"The GPT-based evaluation uses GPT-4o as a binary judge, but the paper does not report human agreement or a small validation of the judge on a sample. Given that the same model family is also used as a baseline, a brief sanity-check of the judge would strengthen the metric.","section":"§4.3"},{"comment":"The table header lists 'PNLS' for all tasks, and the text says PNLS is computed between model answers and ground truths, but the exact normalization and partial alignment procedure is only referenced to [43]. A one-sentence definition or formula would make the metric self-contained.","section":"§4.1, Table 1"},{"comment":"The phrase '1.3 million bars / measures' is inconsistent with the reported 10–20 bars per sheet and 96k sheets (which would give 0.96M–1.92M bars, depending on the count convention); please reconcile the statistics and clarify whether repeats or different clefs are counted separately.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is plausible but currently rests on in-distribution synthetic evaluation, and the dataset-size inconsistencies make it hard to verify the resource. I would encourage a revision that adds a real-sheet or cross-corpus evaluation and resolves the statistics before considering acceptance. The GitHub link in the footnote and the 'code, data, and model will be released upon acceptance' statement are positive signals, but the release promise does not substitute for the missing external validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on MusiXQA. The core artifact is real: a synthetic music sheet VQA dataset generated with MusiXTeX, a new compact note representation (kern+), and a LoRA-fine-tuned Phi-3-Vision that beats GPT-4o on the dataset's OMR and chord tasks. The paper does something useful by showing that output token format matters a lot for structured prediction—the JSON vs. kern+ comparison is the most interesting part—and it documents the generation pipeline thoroughly enough to reproduce. Credit where due: the scale of the synthetic data is large, the annotation is exact by construction, and the zero-shot baselines (Paligemma2, Phi-3-V, GPT-4o) confirm that off-the-shelf MLLMs are near-random on symbolic music tasks.\n\nBut the paper makes claims that the evidence doesn't support. First, there's a direct numerical contradiction: the abstract and intro say 9,600 sheets and 130k QA pairs; Section 3.1 says 96k sheets and 1.34M QA pairs. That's an order of magnitude on both. Until that's fixed, the dataset size is unverifiable. Second, Table 1's caption calls the GPT-4o + RAG + OMR baseline 'oracle OMR,' but the method actually feeds Oemer's noisy, lossy MusicXML into GPT-4o. That's not an oracle, and the label is misleading. Third, and most importantly, the headline gains are all on synthetic sheets from the same pipeline used to generate training data. The authors admit in the Limitation section that the music is chord-heuristic, not real compositions; there is zero evaluation on real engraved, scanned, or handwritten scores. So the phrase 'music sheet understanding' is doing more work than the data allows. The paper is really about synthetic sheet understanding, and should say so.\n\nOther soft spots are minor: no error bars, no human agreement check on the GPT-4o judge, and code/data/models aren't actually released despite the footnote saying so. These are all fixable.\n\nWho is this for? Researchers working on document understanding for MLLMs, music OMR, and structured output decoding. The kern+ finding is worth reading even if you skip the benchmark. As it stands, I would not cite it yet, because the numbers don't add up and the data isn't out.\n\nRecommendation: send to peer review. The problems are fixable, and the contribution—if cleaned up and re-scoped to synthetic sheets—is useful. I'd tell the authors to reconcile the stats, drop the 'oracle' label, and either add a small real-sheet evaluation or firmly limit the claims. Right now the paper is a solid draft, not a finished benchmark.","headline":"Worth a serious look, but the headline claims about music-sheet understanding rest on synthetic-only evidence and a dataset that doesn't add up in the text.","tokens_in":23521,"tokens_out":2970,"would_cite":false,"duration_ms":27102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a small open multimodal model on synthetic sheet music with a compact symbolic note format yields up to eight times the GPT-evaluated accuracy of GPT-4o baselines on OMR and six times on chord estimation.","keywords":["multimodal large language models","sheet music understanding","optical music recognition","visual question answering","synthetic data generation","parameter-efficient fine-tuning","output representation design","chord estimation"],"falsifier":"Run the released Phi-3-MusiX model on the same OMR and chord QA templates over a held-out set of real published sheet music, both engraved and scanned, that was not generated by MusiXTeX; if its GPT-evaluated accuracy on OMR falls to near the single-digit level of the zero-shot GPT-4o baseline rather than staying above 60, the central transfer claim is falsified.","tokens_in":22462,"feed_emoji":"🎼","tokens_out":6070,"duration_ms":61788,"temperature":0.7,"pith_summary":"This paper claims that a small open multimodal language model can learn to read sheet music when trained on a large, synthetically generated question-answering dataset, and that the text format used for musical notes is decisive. The authors build MusiXQA, a benchmark of MusiXTeX-rendered sheets with ground-truth annotations for note pitch and duration, chords, clefs, key and time signatures, titles, and layout, and show that current frontier models such as GPT-4o perform near randomly on OMR and chord estimation. They then fine-tune Phi-3-Vision with LoRA to produce Phi-3-MusiX; using a compact symbolic representation called kern+, it reaches 68.4 GPT-accuracy on OMR and 84.9 on chord estimation, roughly eight times and six times the strongest GPT-4o baseline respectively. The same model trained to emit verbose JSON does much worse, supporting the claim that representation format, not just scale or supervision, drives structured prediction quality.","feed_headline":"Compact note tokens give a small model 8x GPT-4o on sheet music","feed_subtitle":"Fine-tuning on 96k synthetic sheets with kern+ beats GPT-4o and shows output format is the real lever.","key_machinery":"The central machinery is a closed loop of synthetic generation and compact symbolic output. MusiXTeX renders randomly sampled, theory-constrained configurations (clefs, key and time signatures, tempo, chord labels, 10 to 20 bars) into high-resolution images, and the QA templates convert ground truth from the LaTeX source directly into questions. On the output side, kern+, a compact extension of the **kern notation where a note is written like qC4 (duration symbol followed by pitch name with octave), carries the argument by making content tokens dominate format tokens. The paper attributes the JSON model's premature convergence and near-random chord performance to format-token dominance.","core_discovery":"MusiXQA is offered as the first large-scale, balanced, synthetic benchmark for visual music-sheet understanding, and Phi-3-MusiX as the first MLLM specifically fine-tuned for it. The paper's central discovery is that a compact, content-only symbolic encoding of notes is what unlocks accurate structured recognition: with kern+, the fine-tuned model reaches 68.4 GPT-evaluated accuracy on OMR and 84.9 on chord estimation, versus 9.2 and 19.6 for the same model trained to output JSON, and versus 8.4 and 13.0 for the best GPT-4o pipeline that even receives oracle OMR text. The explanation offered is token efficiency: JSON spends most of its token budget on punctuation and keys, so the model learns format before content, while kern+ forces the loss to concentrate on pitches and durations.","pith_inferences":["If the representation-format result generalizes, it suggests that MLLM structured-output failures often diagnosed as reasoning errors may actually be tokenization errors: JSON-like formats let the model learn braces and keys without learning content.","A natural next test is to measure Phi-3-MusiX on real engraved, scanned, or handwritten sheets; the paper does not do this, and the synthetic-to-real gap is the main open question.","The chord-estimation result hints that the model may be learning harmonic inference rather than only symbol reading; one could probe this by holding out chord types or transposing keys and checking whether accuracy transfers.","The manuscript contains a numeric inconsistency: the abstract reports 9,600 sheets while Section 3.1 reports 96,000; this does not change the mechanism but should be resolved before relying on the scale claims."],"forward_implications":["If the gains hold, a roughly 4-billion-parameter open model fine-tuned on synthetic data can outperform much larger proprietary models on note-level sheet-music extraction, making specialized OCR pipelines unnecessary for clean typeset sheets.","A compact, content-centric output tokenization should be preferred over verbose structured formats for any MLLM task whose correctness hinges on a few precise tokens, such as tables, timestamps, or chemical formulas.","Current GPT-4o-style baselines, even with retrieval and an external OMR module, mostly mirror answer formats rather than recognize symbols, so benchmark scores need semantic metrics like G-Acc, not just string similarity.","Because the training signal is synthetic and template-based, the same pipeline can be scaled to other notation systems, such as guitar tablature, percussion, or historical notations, without manual annotation."],"supporting_citations":[{"why":"MusiXTeX is the rendering engine that turns sampled LaTeX configurations into the sheet-music images that form the dataset.","marker":"[36]"},{"why":"Phi-3 technical report supplies the base model, Phi-3-Vision, from which Phi-3-MusiX is fine-tuned.","marker":"[38]"},{"why":"LoRA is the parameter-efficient adaptation method used to fine-tune the base model on MusiXQA.","marker":"[41]"},{"why":"Oemer is the external OMR pipeline whose MusicXML output is fed to GPT-4o in the RAG+OMR baseline.","marker":"[24]"},{"why":"Sheet Music Transformer provides the image encoder used to build the similarity-based retriever for the GPT-4o RAG baseline.","marker":"[28]"},{"why":"MMR defines the Partial Normalized Levenshtein Similarity metric used alongside GPT accuracy for evaluation.","marker":"[43]"},{"why":"The **kern representation is the source of the duration symbols that kern+ extends with octave-indexed pitch names.","marker":"[42]"},{"why":"MMMU documents that prior MLLMs perform near randomly on system-level music score questions, motivating the new benchmark.","marker":"[2]"}],"fun_headline_variants":["kern+ note encoding gives small model 8x GPT-4o on sheet music","Sheet music benchmark shows compact symbolic tokens trump verbose JSON","MusiXQA: small model with kern+ beats GPT-4o by 8x","Compact note encoding unlocks 8x gain in sheet music AI","Format beats size: kern+ makes small model 8x GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains are measured only on MusiXTeX sheets whose musical content was created by the paper's own chord-based heuristics, so the whole result assumes those synthetic sheets stand in for real sheet music; the Limitation section concedes they are not real compositions.","fun_headline_variants_meta":{"raw":{"variants":["kern+ note encoding gives small model 8x GPT-4o on sheet music","Sheet music benchmark shows compact symbolic tokens trump verbose JSON","MusiXQA: small model with kern+ beats GPT-4o by 8x","Compact note encoding unlocks 8x gain in sheet music AI","Format beats size: kern+ makes small model 8x GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2320,"prompt_tokens":906,"completion_tokens":1414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1313}},"tokens_in":522,"tokens_out":1414,"duration_ms":11776,"temperature":1.0,"reasoning_tokens":1313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:52:18.445089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released Phi-3-MusiX model on the same OMR and chord QA templates over a held-out set of real published sheet music, both engraved and scanned, that was not generated by MusiXTeX; if its GPT-evaluated accuracy on OMR falls to near the single-digit level of the zero-shot GPT-4o baseline rather than staying above 60, the central transfer claim is falsified.","supporting_citations":[{"cited_title":"MusiXTEX","cited_arxiv_id":null,"evidence_quote":"MusiXTeX is the rendering engine that turns sampled LaTeX configurations into the sheet-music images that form the dataset."},{"cited_title":"Breezewhite/oemer: v0.1.7, October 2023","cited_arxiv_id":null,"evidence_quote":"Oemer is the external OMR pipeline whose MusicXML output is fed to GPT-4o in the RAG+OMR baseline."},{"cited_title":"Sheet music transformer: End-to-end optical music recognition beyond monophonic transcription, 2024","cited_arxiv_id":null,"evidence_quote":"Sheet Music Transformer provides the image encoder used to build the similarity-based retriever for the GPT-4o RAG baseline."},{"cited_title":"Music information processing using the humdrum toolkit: Concepts, examples, and lessons","cited_arxiv_id":null,"evidence_quote":"The **kern representation is the source of the duration symbols that kern+ extends with octave-indexed pitch names."}],"review_version":1}