{"id":"81c7eab9-d9fb-4601-9b4d-820cb520e92e","arxiv_id":"2507.04776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A music BERT with bounded token denoising and pianoroll prediction beats standard masked-language pre-training on a new 12-task symbolic music benchmark.","lead":"This paper trains a BERT-like music model with two new pre-training tricks: it corrupts note tokens with small random noise and trains the model to reconstruct them, and it also asks the model to predict pianoroll-style note patterns. The authors test the model on 12 symbolic-music understanding tasks and report that it beats the standard MidiBERT on most of them, while releasing a benchmark and source code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposed objectives' gains over same-architecture baseline are small and unreplicated; without error bars or a Pianoroll-only ablation, the central effectiveness claim is not yet statistically supported.","rationale":"I read the paper as claiming that the two pre-training objectives improve downstream symbolic-music understanding. The intended evidence is the within-M2BERT comparison, which controls architecture and data; this is a real strength of the design. The least secure condition for the central claim is not the paper's downbeat-estimation issue: that tokenization affects all compared models equally, and the pianoroll targets are derived from the same bar boundaries, so it is a robustness concern rather than a confound for the objective comparison. The load-bearing issue is statistical and ablation-based. Table 1 reports single runs, and the differences attributable to the proposed objectives are often very small — on the order of 0.001 to 0.03 — while the only comparison that isolates the pianoroll objective yields negative or zero differences on several tasks. Without error bars or a significance analysis, the claim 'results confirm the effectiveness' is not yet established. The paper has supporting infrastructure: public code, a public benchmark, a unified evaluation protocol, and an honest acknowledgment that VF is a counterexample and that individual objectives do not benefit every task. No logical flaw or deceptive result is apparent. The appropriate verdict is therefore conditional acceptance pending a statistical verification, which matches the reader's CONDITIONAL verdict, so no change to the verdict is needed.","tokens_in":13056,"tokens_out":4939,"duration_ms":56889,"concrete_test":"Rerun the Table 1 comparison (M2BERT MLM, RC4,12,12, and RC4,12,12+Pianoroll on the Reduced dataset) with at least five random seeds, fine-tuning on all 12 downstream tasks, and report per-task means, standard deviations, and paired significance. If the combined objective does not beat MLM on at least 9 of 12 tasks at a reasonable threshold, or if its advantage over RC4,12,12 is not significant, the central claim should be softened to a directional improvement rather than a confirmed effectiveness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the abstract as 'Results confirm the effectiveness of the proposed pre-training objectives on downstream tasks,' rests on within-M2BERT comparisons in Table 1. The cleanest isolate of the two proposed objectives is M2BERT RC4,12,12+Pianoroll versus M2BERT MLM (same architecture, same data, same fine-tuning protocol). The gains are small and nonuniform: e.g., SGC +0.002, BP 0.000, CR +0.022, LK +0.008, ME +0.007, OTC +0.030, ER +0.026, VF +0.027. More tellingly, when RC4,12,12+Pianoroll is compared with RC4,12,12 — the comparison that isolates the pianoroll objective — the differences are mostly within a few thousandths and are negative on 5 of 12 tasks (SGC −0.001, BP 0, DbP −0.011, ME −0.001, VE −0.002, MNID −0.001). No standard deviations, confidence intervals, number of seeds, or significance tests are reported anywhere in Section 6 or Table 1. With single-run numbers at this scale, a plausible null hypothesis is that the objective-specific differences are seed noise. This does not invalidate the method, but it means the paper's headline assertion of improvement is not yet statistically established. A secondary gap is that the pianoroll objective is never trained alone, so its individual contribution cannot be cleanly separated from the token-denoising objective even in principle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M2BERT, a ModernBERT-based symbolic-music model, together with two pre-training objectives: bounded token denoising (RC4,12,12), which corrupts note attributes with limited noise and trains the model to reconstruct them, and pianoroll prediction, which trains the model to predict bar-level and local pianoroll/chroma targets. The authors also introduce the SMC benchmark, comprising 12 symbolic-music classification tasks with a unified fine-tuning protocol, and report experiments comparing MidiBERT, several M2BERT variants, two pre-training corpus sizes, and different maximum sequence lengths. The main empirical claim is that the proposed objectives improve downstream symbolic-music understanding, with the best configuration reaching or exceeding previous SOTA numbers on several tasks.","tokens_in":13386,"tokens_out":8006,"duration_ms":88259,"significance":"If the empirical claim is robust, the paper makes a useful contribution: it offers two principled, music-aware pre-training objectives, a reproducible 12-task benchmark, and a careful set of ablations that separate bounded from unbounded token corruption. The benchmark and the public code/data links are concrete assets for the community. The bounded-denoising idea is a reasonable middle ground between masking and full random replacement, and the sequence-length ablation gives useful practical guidance. However, the central claim is currently supported by single-run numbers without uncertainty estimates, and the pianoroll objective is never evaluated by itself, so the evidence is not yet at the level the abstract claims.","major_comments":[{"comment":"The paper's headline claim, stated in the abstract as \"Results confirm the effectiveness of the proposed pre-training objectives,\" rests on comparisons in Table 1 that are reported without any uncertainty quantification. For the comparison that isolates the pianoroll objective, M2BERT RC4,12,12+Pianoroll versus M2BERT RC4,12,12, the differences are mostly a few thousandths and are negative on four tasks (DbP −0.011, ME −0.001, VE −0.002, MNID −0.001) and zero on BP. Even the combined-objective comparison against the same-architecture MLM baseline shows gains of at most 0.030 on most tasks, and no standard deviations, confidence intervals, number of seeds, or significance tests are provided anywhere in Section 6. At this scale the observed differences could plausibly be seed noise. Please report multiple fine-tuning runs (at least 3–5 seeds) with means and standard deviations, or a paired significance test, and adjust the strength of the conclusions to what the statistics support.","section":"§6, Table 1"},{"comment":"The pianoroll prediction objective is never trained alone. The comparison RC4,12,12+Pianoroll versus RC4,12,12 isolates only the marginal effect of adding pianoroll prediction on top of token denoising; it cannot establish the individual contribution of the pianoroll objective, which the paper presents as one of its two main contributions. A Pianoroll-only condition, such as MLM+Pianoroll or token-denoising-free pianoroll prediction, is needed to determine whether the objective is beneficial by itself or only in interaction with token denoising. Without that condition, the claim that the proposed objectives improve performance is not fully supported.","section":"§3.3.2, Table 1"},{"comment":"Both the tokenization and the pianoroll targets rely critically on downbeat timings DB. Section 3.1 says that for MIDI data, \"an unreliable yet doable way\" is to retrieve tick information, and the pre-training corpus includes performance MIDI such as Pianist8 and ASAP, where such information may be noisy or absent. If the downbeat estimates are wrong, the bar boundaries, the rescaling to four crotchet beats, and the pianoroll/chroma targets all become misaligned with the actual musical content. This directly threatens the mechanism of the pianoroll objective. Please either quantify the reliability of DB estimation on the pre-training corpus, restrict the corpus to reliable downbeat annotations, or demonstrate robustness to small downbeat perturbations, before concluding that the pianoroll objective is effective.","section":"§3.1 and §3.3.2"}],"minor_comments":[{"comment":"The sentence listing tasks where the full model \"outperforms the SOTAs\" is internally inconsistent: it enumerates seven tasks (DbP, CR, LK, ME, OTC, PS, VF) but says \"six of the ten tasks,\" and LK is not actually beaten in Table 1 (0.811 versus SOTA 0.829). Please correct the enumeration and the count.","section":"§6, SOTA discussion"},{"comment":"Section 5 says the model \"randomly corrupts 30% of the note tokens,\" while Section 3.3.1 describes MLM as using 15% of tokens in practice. Please state explicitly whether the 30% ratio applies to all compared conditions, including the MidiBERT MLM baseline and the M2BERT MLM baseline; if it does not, the M2BERT-versus-MidiBERT comparison in Section 6 partly conflates architecture with corruption ratio.","section":"§5"},{"comment":"The notation rand(clip(posn − rpos), clip(posn + rpos)) is slightly informal: it should be made explicit that clip returns an integer endpoint in the valid token range and that the draw is over integers in the closed interval [x, y]. The limiting case rpos = rpit = rdur = ∞ also deserves a short formal definition rather than an appeal to the reader's intuition.","section":"§3.3.1, Eqs. (2)–(4)"},{"comment":"There is a missing space in \"pianoroll predictionobjective,\" and the line break in \"PRn and CMn\" makes the expression harder to read. These are purely typographical issues.","section":"§3.3.2, Eq. (5) and following text"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the venue and the benchmark is a genuine community resource. The main barrier is statistical: the central claim is currently supported by single-run numbers, and the missing Pianoroll-only ablation and missing downbeat-reliability analysis are fixable within the paper's scope. I therefore regard this as a major revision rather than a rejection, provided the authors add the requested experiments or substantially temper the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper contributes two things: a bounded-range token corruption objective (RC4,12,12) that is a genuine variant of MLM for symbolic music, and a pianoroll/chroma prediction auxiliary task. The 12-task benchmark is the most useful part — it is public, reproducible, and more comprehensive than the usual four or five tasks. The ablations are sensibly ordered (MLM vs RC∞ vs RC4,12,12 vs RC4,12,12+Pianoroll), and the authors are upfront that gains are not uniform.\n\nThe soft spots are real but they are about interpretation, not execution. The abstract says 'Results confirm the effectiveness' but no error bars or significance tests appear anywhere. On the cleanest comparison — RC4,12,12+Pianoroll versus MLM under the same M2BERT architecture — most gains are a few thousandths of a point; SGC moves .397 to .405, BP is flat, VF actually drops. Some of the larger deltas (DbP +.019, CR +.022, ME +.007) could be real, but without multiple seeds we can't know. Worse, the pianoroll objective is never trained alone, so its individual contribution cannot be separated from token denoising in principle. Comparing RC4,12,12+Pianoroll to RC4,12,12 shows negative deltas on 5 of 12 tasks, so the evidence that pianoroll prediction helps is thin.\n\nThe downbeat-timing dependency is a real practical caveat but not a fatal flaw; they acknowledge the 'unreliable yet doable' MIDI tick extraction and rescaling. The SOTA comparisons are honest about mismatches in evaluation settings.\n\nI agree with the conditional verdict. The benchmark alone justifies serious refereeing. The method ideas are reasonable and well-motivated with a credible mechanism (pitch intervals). The paper is not incoherent; the analysis is honest about nonuniform gains. The main fix is to add error bars or soften the claim. A pianoroll-only ablation would also help. I'd send it to review, with a request for statistical grounding.","headline":"A useful 12-task benchmark and two reasonable pre-training ideas, but the effectiveness claim outruns the statistics without error bars or a pianoroll-only ablation.","tokens_in":13936,"tokens_out":3383,"would_cite":true,"duration_ms":32515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that bounded token denoising and pianoroll prediction pre-training objectives improve symbolic music understanding in a BERT-style encoder, outperforming MidiBERT on almost all of twelve downstream tasks.","keywords":["symbolic music understanding","pre-trained models","BERT","token denoising","pianoroll prediction","self-supervised learning","music information retrieval","downstream evaluation"],"falsifier":"Pre-train the same model and data but compute the pianoroll targets from randomly permuted downbeat timings; if downstream gains on chord and melody tasks survive, the pianoroll objective is not doing the work the paper claims.","tokens_in":12820,"feed_emoji":"🎼","tokens_out":13182,"duration_ms":109491,"temperature":0.7,"pith_summary":"The paper claims that two self-supervised pre-training objectives—bounded token denoising and pianoroll prediction—make a BERT-style model (M2BERT, built on a 12-layer ModernBERT) better at understanding symbolic music than the standard masked-language-model approach used by MidiBERT. The authors argue that plain MLM treats notes as isolated tokens and never teaches the model the musical relationships between them, such as pitch intervals and metrical structure. Over a new benchmark of 12 downstream classification tasks, the best M2BERT configuration outperforms MidiBERT on almost all tasks, with the largest gains on downbeat prediction, chord root estimation, local key estimation, and melody extraction. If correct, this points to pre-training objectives that encode musical geometry rather than larger architectures alone as the key to stronger symbolic music models.","feed_headline":"Bounded token noise plus pianoroll targets lift BERT on 12 music tasks","feed_subtitle":"By predicting pianorolls from corrupted tokens, the model learns musical structure without task-specific engineering.","key_machinery":"The two pre-training objectives. Token denoising corrupts a note token's attributes by bounded random noise—onset position within ±4, pitch within ±12 semitones, duration within ±12 units—and trains the encoder with cross-entropy loss to recover the original attributes, with the bound serving as the mechanism that forces the model to reason about near-miss values and thereby learn interval structure. Pianoroll prediction adds an L2 loss that requires each note embedding to predict a 16-by-86 pianoroll and a 16-by-12 chromagram for its bar (bar-level) and for its own onset time (local), directly injecting the simultaneous-pitch layout of the music into the learned representation. Both objectives share the same note-level contextualized encoder and prediction machinery, so the pre-training signal is a sum of token reconstruction loss and pianoroll prediction loss.","core_discovery":"The paper's central claim is that replacing masked language modeling with two music-aware objectives makes a BERT-style encoder better at symbolic music understanding. In the token denoising objective, roughly 30% of note tokens have their onset position, pitch, and duration attributes perturbed within bounded ranges (position by ±4, pitch and duration by ±12), and the model learns to reconstruct the original attribute values, which teaches the relative distances between token values that carry musical meaning such as pitch intervals. In the pianoroll prediction objective, each note's embedding is additionally trained with an L2 loss to reproduce the pianoroll and chromagram of its enclosing bar and of its own onset position, injecting explicit information about which pitches sound together and when. Combined into a 12-layer ModernBERT backbone and pre-trained on up to 350 million notes, the resulting M2BERT model outperforms MidiBERT on almost all twelve downstream tasks, with the largest gains on downbeat prediction, chord root estimation, local key estimation, and melody extraction.","pith_inferences":["The bounded-denoisng principle should transfer to other structured-sequence domains whose tokens live on an ordered metric, where corrupting within a limited radius and denoising would teach distance structure analogous to musical intervals.","Since the pianoroll target is computed from downbeats, coupling the model with a learned beat tracker, or treating the downbeat as a predicted variable, would be a natural robustness extension and a direct stress test of the objective.","The SMC benchmark's focus on classification tasks may under-represent what the objectives do for generation or score-to-performance mapping; those settings could show different trade-offs.","The concentration of gains in pitch-interval-sensitive tasks yields a testable prediction: corrupting only duration and onset while keeping pitch intact should not reproduce the chord and melody improvements."],"forward_implications":["Bounded denoising (RC4,12,12) outperforms both standard MLM and unbounded random replacement (RC∞), showing that limiting the corruption range is itself a source of the improvement.","Adding the pianoroll head yields the largest gains on tasks that depend on pitch, interval, and harmonic structure, such as chord root, local key, melody, texture, and downbeat prediction, while leaving beat-note prediction and velocity estimation roughly unchanged.","Scaling the pre-training corpus from 4.89 million to 350.32 million notes improves nearly every downstream task, and increasing the input context length from 512 to 2048 tokens helps most on chord and key estimation.","The best configuration reaches or beats previously reported task-specific state-of-the-art results on several tasks without task-specific architectures, data augmentation, or multi-task learning."],"supporting_citations":[{"why":"Defines the MidiBERT baseline that M2BERT compares against, including the CP tokenization, fine-tuning protocol, and pre-training data.","marker":"[1]"},{"why":"Supplies the BERT architecture and masked language modeling objective that the paper modifies.","marker":"[11]"},{"why":"Provides the Compound Word token representation (bar, position, pitch, duration) used to encode notes.","marker":"[42]"},{"why":"Provides the ModernBERT backbone that M2BERT adapts to 12 layers.","marker":"[44]"},{"why":"Gives the argument that [MASK] tokens create a pre-training/fine-tuning distribution discrepancy, motivating token denoising.","marker":"[47]"},{"why":"Supplies the PM2S dataset and evaluation setup for the beat and downbeat note prediction tasks.","marker":"[48]"},{"why":"Provides the segmentation strategy with a 1024-token maximum sequence length.","marker":"[5]"},{"why":"Supplies the AugmentedNet dataset and split used for chord root estimation and local key estimation.","marker":"[15]"},{"why":"Supplies the POP909 dataset used for melody extraction evaluation and as a pre-training corpus.","marker":"[49]"},{"why":"Supplies the EMOPIA dataset used for emotion recognition and as a pre-training corpus.","marker":"[25]"}],"fun_headline_variants":["Bounded note noise and pianoroll goals boost BERT on 12 tasks","Denoising tokens and predicting pianorolls lifts BERT for music","Fix notes and see piano rolls: new BERT training wins on music","Predicting pianorolls from noisy tokens sharpens music BERT","Two music-aware objectives help BERT excel on symbolic tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that downbeat timings derived from MIDI tick information are correct enough that the pianoroll and chroma targets align with the actual music, because those targets are computed from bars defined by the downbeats.","fun_headline_variants_meta":{"raw":{"variants":["Bounded note noise and pianoroll goals boost BERT on 12 tasks","Denoising tokens and predicting pianorolls lifts BERT for music","Fix notes and see piano rolls: new BERT training wins on music","Predicting pianorolls from noisy tokens sharpens music BERT","Two music-aware objectives help BERT excel on symbolic tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00111,"raw_usage":{"total_tokens":4596,"prompt_tokens":885,"completion_tokens":3711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":3617}},"tokens_in":501,"tokens_out":3711,"duration_ms":28726,"temperature":1.0,"reasoning_tokens":3617,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:38:53.665431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train the same model and data but compute the pianoroll targets from randomly permuted downbeat timings; if downstream gains on chord and melody tasks survive, the pianoroll objective is not doing the work the paper claims.","supporting_citations":[{"cited_title":"Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Prediction","cited_arxiv_id":"2507.04776","evidence_quote":"Defines the MidiBERT baseline that M2BERT compares against, including the CP tokenization, fine-tuning protocol, and pre-training data."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the ModernBERT backbone that M2BERT adapts to 12 layers."},{"cited_title":"Dynamic-SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,","cited_arxiv_id":null,"evidence_quote":"Gives the argument that [MASK] tokens create a pre-training/fine-tuning distribution discrepancy, motivating token denoising."},{"cited_title":"MERT: acoustic music understanding model with large-scale self-supervised training,","cited_arxiv_id":null,"evidence_quote":"Supplies the PM2S dataset and evaluation setup for the beat and downbeat note prediction tasks."},{"cited_title":"no label","cited_arxiv_id":null,"evidence_quote":"Provides the segmentation strategy with a 1024-token maximum sequence length."},{"cited_title":"MusicBERT: Symbolic music understanding with large-scale pre-training,","cited_arxiv_id":null,"evidence_quote":"Supplies the AugmentedNet dataset and split used for chord root estimation and local key estimation."},{"cited_title":"Music Foundation Model as Generic Booster for Music Downstream Tasks","cited_arxiv_id":"2411.01135","evidence_quote":"Supplies the POP909 dataset used for melody extraction evaluation and as a pre-training corpus."},{"cited_title":"Augment- edNet: A roman numeral analysis network with syn- thetic training examples and additional tonal tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the EMOPIA dataset used for emotion recognition and as a pre-training corpus."}],"review_version":1}