{"id":"896a43f2-8701-4f0b-9b76-687805fd7776","arxiv_id":"2607.08168","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MuScriptor—a decoder-only transformer combining synthetic pre-training, real-data fine-tuning, and GRPO post-training—reaches 48.2 Multi-F1 on the authors' internal 372-track test set versus 21.9 for YourMT3+.","lead":"This paper releases MuScriptor, an open-weight model that transcribes multi-instrument music audio into MIDI, trained with synthetic-data pre-training, fine-tuning on 11k hours of real recordings, and reinforcement-learning post-training. It also analyzes how much synthetic pre-training helps and lets users condition on which instruments are present to stabilize and customize transcriptions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline results hinge on unvalidated note-level accuracy of automatically synchronized D_Real/D_Test annotations; alignment bias could inflate all D_Test numbers and explain the Bach10 onset regression.","rationale":"The reader identified the alignment quality of D_Real/D_Test as the weakest assumption, and I agree. This concern is the most load-bearing because it attacks the foundation of the empirical evaluation: if the ground truth itself is biased, all reported F1 scores and the headline conclusion are unreliable. The paper provides no external validation of the synchronization pipeline's note-level accuracy, and the Bach10 regression is a red flag that the model's onset predictions do not transfer to cleanly annotated data. The concrete test—an expert audit of D_Test—directly addresses this by measuring whether the ground truth meets the 50 ms onset tolerance that the metrics assume. Even if the audit passes, the oracle conditioning issue should be reported separately, but it is secondary. The reader's CONDITIONAL verdict is appropriate: the paper's contributions (open weights, dataset, recipe) stand, but the quantitative claims should be accepted only after the annotation quality is verified. Thus I recommend no change to the verdict.","tokens_in":11080,"tokens_out":4015,"duration_ms":40513,"concrete_test":"Select 50 random tracks from D_Test. Have two expert annotators independently transcribe note onsets/offsets for a 30-second excerpt of each track (or use a state-of-the-art score follower as a cross-check). Compute note-level onset (and offset) agreement between the D_Test ground-truth annotations and the manual/expert reference using the same mir_eval metrics. If the median onset error > 50 ms or the F1 between automated and manual annotations is < 90%, the D_Test ground truth is too noisy to support Table 1's claims. Additionally, re-run the MuScriptor vs YourMT3+ comparison on the subset of D_Test with high annotation agreement, and without instrument conditioning, to see whether the 48.2 vs 21.9 margin holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that MuScriptor is the first effective open-weight model for multi-instrument transcription—rests on Table 1, evaluated on D_Test. Both D_Real (training) and D_Test (test) annotations are derived from audio-symbolic synchronization (§3.1): linear interpolation between bar lines or DTW with chroma/onset features. The filtering thresholds (warping distance, time-dilation factor) only ensure global alignment plausibility, not note-level accuracy. The D_Test selection criterion ('highly accurate annotations') is self-referential: it comes from the same unvalidated pipeline. If alignment errors are systematic—e.g., biased onset times or missed notes in dense mixes—then the training targets and test ground truth are both corrupted, and the reported gains over YourMT3+ (Multi F1 48.2 vs 21.9) may not reflect real-world transcription quality. The unexplained Bach10 regression (onset F1 43.1 vs 59.8 for YourMT3+) is consistent with this: models trained on misaligned onsets may generalize poorly to precisely annotated datasets. A secondary concern is that Table 1 uses oracle instrument conditioning (ground-truth instrument lists), which the baseline lacks; though the ablation shows only a ~2-point effect, it still confounds the headline comparison. The alignment concern is the most load-bearing because it undermines the evaluation itself, not just the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MuScriptor, a decoder-only transformer for multi-instrument music transcription that is trained on 1.45M synthetic MIDI files and 170k real recordings (11k hours) with audio-symbolic alignments, then post-trained with a GRPO-style reinforcement learning objective using 300 manually verified tracks. The model optionally conditions on instrument presence, and the authors release model weights and inference code. The central empirical claim is that MuScriptor is the first effective open-weight multi-instrument transcription model, supported by large gains over YourMT3+ on a curated in-house test set D_Test (e.g., Multi F1 48.2 vs 21.9) and by improvements on several external benchmarks.","tokens_in":11367,"tokens_out":7491,"duration_ms":74534,"significance":"If the central claim holds, this is a significant contribution: the open-weight model, the data recipe combining synthetic pre-training with large-scale real fine-tuning, the RL post-training, and the instrument conditioning are all useful to the AMT community. The internal ablations are consistent, the synthetic-data analysis is informative, and the model release is a concrete asset. However, the main evaluation depends on automatically synchronized annotations that are shared between training and test, and several independent benchmarks show regressions relative to the baseline. The paper's value is therefore conditional on validating the alignment pipeline and on presenting the D_Test results as in-domain rather than as evidence of general-purpose transcription quality.","major_comments":[{"comment":"The headline comparison is evaluated on D_Test, which is selected from D_Real, whose annotations are produced by audio-symbolic synchronization (linear bar-line interpolation or DTW with chroma/onset features). The stated filters only constrain global warping distance and a maximum time-dilation factor; they do not validate note-level accuracy. If synchronization errors are systematic, both training targets and D_Test ground truth are corrupted in a correlated way, so the reported 48.2 vs 21.9 Multi F1 gap can overstate real-world performance. The independent benchmarks in Table 2 are mixed: Bach10 onset F1 drops from 59.8 (YourMT3+) to 43.1, and RWC-C/RWC-R also show regressions. Please provide a quantitative validation of the alignment pipeline (e.g., note-level agreement on a manually annotated sample), state whether D_Test was manually verified, and report per-genre and per-dataset i","section":"§3.1, Table 1"},{"comment":"It is not stated whether D_RL and D_Test are disjoint. D_RL is described as 300 tracks selected from D_Real, and D_Test as 372 tracks subsequently selected from D_Real; the text says only that D_Test tracks are removed from D_Real and that similar titles are removed from training sets. If D_RL tracks were not excluded from the D_Test selection, the RL post-training set could overlap with the test set, inflating the D_RL row of Table 1. Please state explicitly that D_RL and D_Test are disjoint, and describe how overlap was checked beyond title matching.","section":"§3.1 (D_RL/D_Test disjointness)"},{"comment":"The headline MuScriptor results use ground-truth instrument conditioning (the actual instruments present in the track), while the YourMT3+ baseline is not given this oracle information. Table 3 shows conditioning contributes about 1.8 Multi F1 on a D_Real-only model, so it does not explain most of the Table 1 gap, but it is still a confound in the claim of being the first effective open-weight model. Please report the main comparison with MuScriptor evaluated without oracle conditioning (using conditioning dropout or predicted instruments), and, if possible, include an instrument-conditioned baseline.","section":"Tables 1 and 3"},{"comment":"The primary D_Test metrics are computed after deleting overlapping same-pitch same-instrument notes that the tokenizer cannot represent. The authors disclose this and report that keeping these notes lowers Multi F1 from 48.2 to 42.0 (α_CFG=1). Since such overlaps are common in real mixes, the headline number is not a direct measure of real-world transcription quality. The conclusion that MuScriptor is 'effective' for general-purpose transcription should be qualified, and the central claims should be presented on both the filtered and unfiltered test sets, not only on the filtered one.","section":"§3.3, §4.2.3"}],"minor_comments":[{"comment":"No error bars or multiple runs are reported for the central numbers. Given the D_Test size and the stochasticity of RL post-training, bootstrap confidence intervals or results across seeds would materially strengthen the comparison.","section":"Table 1 and Figure 4"},{"comment":"The notation L_CE(θ, τ; y) uses l_θ(y_t), which is called a logit, but the expression is a temperature-scaled cross-entropy term. Clarify that this is a token-level loss and how it relates to the REINFORCE estimator's log-probability.","section":"Eq. (3)"},{"comment":"The sentence attributing lower onset/offset scores to 'difficulty of annotating precise note on- and offsets for certain styles' does not explain the Bach10 onset regression. A specific analysis of the Bach10 case would be informative.","section":"§4.1.2"},{"comment":"The paper is titled 'Open Model' and releases weights and inference code, but D_Real and D_Test are internal/commercial and not released. Please document licenses and consider releasing a small public evaluation subset to make the benchmark reproducible.","section":"§3.1 / Data release"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and likely to be highly cited if the data pipeline can be validated. The most important issue is not the ablations but the self-referential nature of D_Test and the unclear D_RL/D_Test disjointness; both are fixable with additional experiments and explicit statements. The external benchmark regressions do not disprove the central claim, but they prevent acceptance in the current form. Please also check the 'first effective open weight model' claim carefully against concurrent work before final publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: MuScriptor is a genuinely useful open-weight multi-instrument transcriber, and the central message—that 11k hours of real aligned data plus synthetic pre-training plus RL post-training beats the MT3-family recipe by a wide margin—holds up internally. The 48.2 vs 21.9 Multi-F1 gap on D_Test is large and consistent with the ablations, but that test set is curated from the same audio-symbolic synchronization pipeline that produced the training annotations. I'd treat the absolute numbers as provisional until independently annotated data confirms them.\n\nWhat's genuinely new and useful: a controlled study of synthetic pre-training at real scale (Fig. 4 is the strongest part—clear diminishing returns, big gains when real data is scarce), the first GRPO-style post-training for transcription, instrument conditioning, and the weight release. The ablations are consistent and the small-model results are useful for low-resource settings. Credit where due: the authors disclose the overlapping-note cleanup and quantify its cost (Multi-F1 drops from 48.2 to 42.0 if you keep those notes). That's honest reporting.\n\nSoft spots, in order:\n\nFirst, the D_Test self-reference. D_Test is selected from D_Real using a 'highly accurate annotations' criterion that depends on the same alignment pipeline used for training. Filtering on warping distance and time-dilation plausibility does not guarantee note-level accuracy. If alignment errors are systematic—say, onset blur in dense mixes—both training and test are biased. The Bach10 onset regression (43.1 for MuScriptor vs 59.8 for YourMT3+) is consistent with that, though frame and multi-F1 improve there. This is the load-bearing concern; it doesn't invalidate the paper but it does mean the headline gap needs an external check.\n\nSecond, single run, no error bars. At 1.3B parameters and 372 test tracks, the margin is probably real but we can't see variance. Easy to fix with multiple seeds.\n\nThird, no MIROS comparison. They cite it and don't evaluate it, which weakens the 'first effective open model' claim.\n\nFourth, oracle instrument conditioning in Table 1: the model receives the ground-truth instrument list. The ablation shows ~2 points Multi-F1, so minor, but still an advantage the baseline doesn't have.\n\nWho this is for: MIR researchers who need a transcription front-end, and generative music people. The weights are out, the recipe is reproducible, and the architecture is boring—which is a plus. It deserves a serious referee. My recommendation: send it out. Require a MIROS comparison, error bars, and at least one independently annotated external corpus (Bach10 is already there; explain the onset regression). The alignment-pipeline audit is the key request, not a blocker.\n\nI'd bring this to a reading group and I'd cite the synthetic-vs-real analysis.","headline":"Solid open-weight transcription model with real data at scale; the headline 48.2 Multi-F1 is provisional because the test set shares the alignment pipeline with the training data.","tokens_in":11923,"tokens_out":3274,"would_cite":true,"duration_ms":29785,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a decoder-only transformer trained on real music audio with aligned note annotations, after synthetic pre-training and reinforcement-learning post-training, can transcribe multi-instrument recordings across genres wit","keywords":["automatic music transcription","multi-instrument","open-weight model","synthetic data pre-training","reinforcement learning post-training","instrument conditioning","decoder-only transformer"],"falsifier":"Have independent musicians hand-transcribe a random sample of test tracks and compare MuScriptor's output against those human labels; if the multi-instrument F1 falls materially below the reported 48.2, the claimed gains are inflated by alignment bias in the reference annotations.","tokens_in":10920,"feed_emoji":"🎵","tokens_out":3563,"duration_ms":32948,"temperature":0.7,"pith_summary":"The authors set out to show that synthetic data alone is not enough for general-purpose music transcription, but a training recipe of large-scale synthetic pre-training, fine-tuning on 11,000 hours of real recordings, and a policy-gradient post-training step produces a model that far exceeds prior baselines. Each stage contributes: synthetic data helps when real data is scarce, real data drives the largest gains, and reinforcement learning on a small high-quality set adds further improvement. The paper also introduces instrument-presence conditioning, which lets users customize transcriptions and stabilizes segment predictions. If the claims hold, the music information retrieval community gets both a usable open-weight transcriber and a clear data-and-training recipe for building one.","feed_headline":"Open model transcribes real music mixes with 2x accuracy","feed_subtitle":"Synthetic pre-training, 11k hours of aligned real audio, and RL post-training push multi-instrument F1 from 21.9 to 48.2.","key_machinery":"The key machinery is the training recipe rather than a new architecture: a plain decoder-only transformer over mel-spectrogram input and MIDI-like event tokens, pre-trained on synthetic audio rendered from 1.45M MIDI files, fine-tuned on 170k real recordings aligned to note annotations via audio-score synchronization, and post-trained with a reinforcement-learning objective that rewards note-level F1. A learned instrument-conditioning prefix lets the user specify which instruments to transcribe and stabilizes segment predictions.","core_discovery":"MuScriptor is a 1.3B-parameter decoder-only transformer that maps a mel-spectrogram to a MIDI-like token sequence, optionally conditioned on which instruments are present. Trained on 1.45 million synthetic MIDI files, then 170,000 real recordings with synchronized note annotations, then refined with a group-relative policy-gradient objective on 300 manually verified high-quality pieces, it reaches a multi-instrument F1 of 48.2, onset F1 of 54.4, and frame F1 of 73.3 on the authors' test set—more than doubling the multi-instrument F1 of the strongest prior baseline. The paper's central claim is that this combination of synthetic pre-training, real-data fine-tuning, and reinforcement-learning","pith_inferences":["The reported numbers rest on a test set built from the same audio-score synchronization pipeline used for training; a fully independent, human-transcribed evaluation set could yield lower estimates, especially for onset and offset precision, noting that the Bach10 onset score actually dropped from 59.8 to 43.1 relative to the baseline, a hint of style-dependent alignment issues.","If the synchronization quality scales with data, further improvements in alignment—or larger real aligned corpora—could push multi-instrument transcription well beyond the plateau reported here.","The instrument-conditioning mechanism could be used interactively, letting a user toggle instruments on and off to separate a track by ear, effectively turning the model into a source-separation-style tool despite not being explicitly trained for separation."],"forward_implications":["If the recipe transfers, other music information retrieval tasks such as chord, key, and melody extraction could adopt the same data-scale strategy instead of relying on architectural tweaks.","The 11,000-hour real aligned dataset becomes a reusable public resource that lowers the barrier for future transcription research.","Instrument-conditioned transcription enables user-facing tools that extract specific instruments, such as bass or vocal lines, from a full mix.","Reinforcement-learning post-training opens a new axis for improving transcription beyond standard supervised teacher forcing."],"fun_headline_variants":["Open model doubles multi-instrument transcription F1","MuScriptor: 2x multi-instrument F1 on real mixes","RL post-training doubles open transcription accuracy","From synthetic to real: open model doubles F1","Open model: multi-instrument transcription for real mixes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the audio-to-note alignment used to build both training and test data is note-accurate across all genres; if alignment errors correlate with musical style, the measured gains partly reflect noise in the reference labels.","fun_headline_variants_meta":{"raw":{"variants":["Open model doubles multi-instrument transcription F1","MuScriptor: 2x multi-instrument F1 on real mixes","RL post-training doubles open transcription accuracy","From synthetic to real: open model doubles F1","Open model: multi-instrument transcription for real mixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000576,"raw_usage":{"total_tokens":2510,"prompt_tokens":656,"completion_tokens":1854,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":1777}},"tokens_in":400,"tokens_out":1854,"duration_ms":13343,"temperature":1.0,"reasoning_tokens":1777,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:23:15.367946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent musicians hand-transcribe a random sample of test tracks and compare MuScriptor's output against those human labels; if the multi-instrument F1 falls materially below the reported 48.2, the claimed gains are inflated by alignment bias in the reference annotations.","supporting_citations":[],"review_version":2}