REVIEW 4 major objections 5 minor
Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A score-predicting transformer that uses a frozen pretrained music encoder and pitch/time augmentation transcribes real popular-music audio into lead sheets at a 20.92% symbol error rate, and cuts classical string-quartet error from 15.3%…
desk verdict A genuinely useful popular-music A2S dataset and a competent model paper; the dataset is the real contribution, and the headline popular benchmark needs more validation before being treated as canonical. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object that carries the argument is MuQ, a self-supervised music-audio feature extractor trained with mel residual vector quantization on a large music corpus; it is kept frozen and its 1024-dimensional output is projected and normalised down to 256 dimensions before serving as cross-attention memory for an autoregressive transformer decoder. The decoder itself is widened (feedforward 256 to 1024) and switched to pre-norm stabilisation, and training adds six pitch-shifted variants per clip (each with a randomly sampled time stretch between 0.9 and 1.1), transposing the symbolic target to match. On the dataset side the load-bearing step is octave inference: vocal stems are separated, a vocal pitch estimator produces per-frame pitch probabilities, twelve octave/start-time hypotheses are scored by normalised probability mass inside note masks, and clips scoring below 0.2 are discarded.
What would settle it
Run an independent octave check on a random sample of the 9,468 retained SheetSage-A2S clips: have trained musicians annotate the sung melody's absolute octave from the audio alone, and compare with the octave assigned by the 12-hypothesis confidence procedure. If the two disagree on more than a few percent of clips that passed the 0.2 threshold, the 20.92% SER benchmark is contaminated by annotation error and would need to be recomputed on a re-cleaned subset.
Extended reading notes
Core claim
The paper's central claim is that direct audio-to-score transcription becomes substantially more accurate when a frozen self-supervised music representation replaces a learned-from-scratch spectrogram encoder, and when pitch/time augmentation is used during training. On the paper's own numbers, the recipe reduces Quartets SER from 15.3% to 4.98%, with the largest single gain on the popular-music dataset coming from the pretrained encoder alone (53.7% to 25.39% before augmentation). The new dataset, SheetSage-A2S, is presented as the first popular-music A2S benchmark, built from real recordings and user-generated lead sheets, and the reported 20.92% SER is offered as a strong baseline for future work rather than a solved problem.
Load-bearing premise
The dataset's ground-truth scores stand or fall on the octave-inference step: for each clip the absolute octave is taken from the highest-scoring of 12 pitch hypotheses, and clips with confidence below 0.2 are discarded—so if that threshold admits a substantial share of wrong-octave clips, the reported 20.92% popular-music SER measures the annotation pipeline as much as the model.
Editorial extensions
If this is right
- Future A2S work on popular music gets a fixed comparison point: 20.92% SER on SheetSage-A2S, with separate melody (38.62%) and chord (22.28%) scores.
- The classical-music result transfers: the same model cuts Quartets SER from 15.3% to 4.98%, a 67.5% relative reduction.
- Because MuQ is frozen, the trainable parts are the projection layer, positional encodings, and decoder, so the approach is relatively cheap to reproduce.
- Word-level tokenisation wins over mid-level and character-level on this task, so sequence-length reduction is worth more here than the bijective token-to-symbol property of mid-level schemes.
- The per-spine numbers indicate chords are the harder subproblem on lead sheets, pointing future work toward chord-specific modelling or cleaned chord annotations.
Reading between the lines
- Because MuQ was pretrained largely on popular music, a portion of the SheetSage-A2S gain may reflect overlap with the pretraining corpus; the paper acknowledges this is hard to quantify, and a clean test is to split the test set into songs likely and unlikely to be in that corpus and compare SER on each.
- Augmentation and the frozen-encoder recipe should transfer to other A2S targets such as instrumental lead melodies or full polyphonic scores; the paper's own future-work section points in that direction, where octave inference would need to rely on a different pitch source than vocal stems.
- The confidence threshold of 0.2 was calibrated on manual inspection of about 150 clips; if the dataset grows, publishing per-clip confidence scores would let downstream users weight or filter samples and re-estimate the benchmark without re-listening.
- Since the model's timing errors are mostly within an eighth note and preserve overall phrase length, a musically-weighted evaluation that tolerates small offset shifts would likely report a smaller effective error than raw SER; the paper notes that existing musically-weighted metrics do not support lead-sheet formats.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SheetSage-A2S, a new popular-music audio-to-score dataset of 9,468 clips (61 hours) paired with **kern lead-sheet annotations derived from SheetSage JSON and YouTube audio, together with an octave-inference pipeline based on RMVPE pitch probabilities. The authors also propose an A2S model that augments the transformer decoder of Alfaro-Contreras et al. with a larger pre-norm decoder, a frozen MuQ encoder, and pitch-shift/time-stretch data augmentation. They report a 4.98% symbol error rate on the Quartets classical corpus versus 15.3% for the prior state of the art, and 20.92% SER on SheetSage-A2S, which they present as a first benchmark for popular-music A2S. An ablation table shows progressive gains from each modification, and a tokenisation study compares character-, mid-, and word-level schemes.
Significance. If the dataset and its ground-truth construction are reliable, SheetSage-A2S is a valuable contribution: it is the first large-scale A2S resource for popular music built from real commercial recordings, it is released with annotations and code, and it addresses a clear gap in the field. The architectural choices (using a frozen pretrained music encoder and simple data augmentation) are well motivated, and the ablation ordering in Table 2 is internally consistent. The paper also provides per-spine melody/chord SER, which is useful for error analysis. However, the central quantitative claims rest on two points that need strengthening: the validation of the octave-inference step that determines the absolute pitches in the new dataset, and the absence of any statistical uncertainty around the reported SER values.
major comments (4)
- [Section 2.2, Octave Inference] The quality check for the ground-truth construction is not sufficiently documented. The retention threshold C > 0.2 is justified by manual inspection of "around 150 clips" and the claim that no incorrect octave was found above the threshold, yet the retained set contains 9,468 clips and the rejected set 6,842 clips. No confidence distribution, no margin between the best and second-best hypothesis, and no independent verification on a held-out random sample of retained clips are reported. Since Figure 4b shows that 98.66% of retained clips are assigned to octaves 3 or 4, a small systematic bias in the octave-inference prior would directly corrupt training targets and the 20.92% SER benchmark. Please add a quantitative validation of retained-set octaves, such as manual verification of a random sample, agreement with an independent pitch tracker, and a report of the confidence margins.
- [Table 2 and Section 5.1] All SER values in Table 2 and Table 3 are point estimates from single training runs, with no repeated seeds, confidence intervals, or significance tests. The claims that the method "significantly outperforms" the prior 15.3% baseline and that each ablation step helps rest entirely on these point estimates. Because neural training is stochastic and the augmentation expansion changes the training data distribution, please provide variance estimates (e.g., 3-5 seeds, or bootstrap over test clips) for the final results and at least the key ablation rows.
- [Section 4.2, Training setup and Table 2] The comparison with the baseline of [1] is not fully specified. The paper states that the training setup differs from the original (AdamW, weight decay 1e-3, lr 5e-5, batch size 8 versus Adam, no weight decay, lr 1e-4, batch size 1), but Table 2 lists "Baseline [1]" as 15.3 on Quartets. Please state explicitly whether this number was reproduced under the new optimizer or copied from [1]. If it was copied, the 4.98% result conflates architectural changes with training-scheme changes, and a retrained baseline under the new scheme is needed for a fair ablation.
- [Section 5.2, Encoder and augmentation] The paper acknowledges that SheetSage-A2S songs may overlap with the MSD corpus used to pretrain MuQ, but it does not quantify this overlap. Because the popular-music benchmark is a central contribution, this overlap could inflate both the apparent benefit of MuQ and the absolute 20.92% SER figure. Please attempt to quantify the overlap (e.g., via metadata matching or audio fingerprints) and report results separately for likely-overlapping and non-overlapping test clips, or provide a justification for why overlap cannot affect the frozen-feature evaluation.
minor comments (5)
- [Section 2.2, Hypothesis selection] The phrase "we discard clips with confidence C = argmax_H C(H) below a threshold of 0.2" is notationally confused: the scalar confidence should be max_H C(H), not the argmax. Please correct the notation.
- [Section 3.1 and Section 5.2] The "1024-PreNorm baseline" changes two architectural factors simultaneously (feedforward dimension and normalisation order). The statement in Section 5.2 that the improvement is "primarily" due to the increased feedforward dimension is not supported by an ablation isolating these two changes. Please add such an ablation or soften the attribution.
- [Figure 4(a)] The histogram is truncated at 100 seconds while the text states that 12 clips exceed 100 seconds. Please report the maximum clip duration and consider showing the tail explicitly (e.g., with an inset).
- [Section 4.1] The artist-stratified split is described for SheetSage-A2S, but since multiple clips can come from one song, please confirm explicitly that all clips of a given song are kept within the same split.
- [Section 6.2] The sentence "Measures 5 to 7 from Love Live - Bokutachi wa Hitotsu no Hikari. 12 It demonstrates..." has an ambiguous footnote marker placement; please clarify the footnote or citation formatting.
Circularity Check
No significant circularity: reported SERs are measured on held-out test sets and the model does not use RMVPE at inference; the octave-inference step is a data-cleaning procedure, not a derived prediction.
full rationale
The paper's derivation chain is self-contained. The core claims are (i) a 4.98% SER on the existing synthetic Quartets benchmark against the 15.3% reported baseline [1], and (ii) a 20.92% SER on the new SheetSage-A2S dataset. Both numbers are measured on held-out test splits with greedy decoding; they are not derived from the input by construction. The new dataset is built from SheetSage JSON annotations and YouTube audio, with absolute octaves inferred by an RMVPE-based hypothesis-selection procedure. This is a label-generation step, and the paper explicitly checks it on ~150 clips. The downstream A2S model does not use RMVPE or the octave-inference confidence at inference time, so the SER is not fitted to, nor a restatement of, the octave-inference output. The paper's own limitations (Section 6.1 and the Section 2.2 footnote about pitch tools tested on synthetic instruments) concern label noise and generalisation risk, not circularity. All references are to external prior work; there are no load-bearing self-citations, uniqueness theorems, or ansatz smuggling. The ablation study in Table 2 provides independent evidence that each architectural modification contributes to the measured SER. No equation or definition reduces a predicted quantity to an input parameter, so no circular step is present.
Assumptions & free parameters
free parameters (6)
- Octave-inference confidence threshold =
0.2
- Silence RMS threshold =
0.0005
- Pitch-shift augmentation range =
{-3,-2,-1,1,2,3} semitones
- Time-stretch augmentation factors =
{0.9,0.95,1.05,1.1}
- Label smoothing =
0.1
- Rhythm quantization maximum denominator =
96
assumptions (6)
- domain assumption SheetSage user annotations are a valid ground truth for melody and chords in popular music.
- domain assumption madmom beat and downbeat tracking reliably maps beat-indexed note onsets to audio timestamps.
- domain assumption RMVPE's normalized pitch probability correctly identifies the absolute octave when the confidence score exceeds 0.2.
- domain assumption Mel-RoFormer vocal separation provides a vocal stem clean enough for RMVPE to estimate melody pitch.
- domain assumption MuQ features learned on the Million Song Dataset transfer to A2S and are not biased by train-test overlap on SheetSage-A2S.
- domain assumption Levenshtein SER on word-level tokens is a meaningful measure of musical score quality.
Cite this review
Pith. "Pith review of Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset." pith.science (2026). https://pith.science/paper/BBDIM4HE
@misc{pith2026260806165,
author = {Pith},
title = {Pith review of: Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/BBDIM4HE}},
note = {Machine review of arXiv:2608.06165}
}
read the original abstract
Existing audio-to-score (A2S) systems primarily focus on classical music, and the application to popular music remains underexplored. This paper first presents the new SheetSage-A2S Dataset, which includes 61 hours of audio with \texttt{**kern} score encodings for 9,468 clips originating from 6,066 unique songs, the first of its kind to facilitate A2S research for popular music. Additionally, we improve on existing A2S approaches by using data augmentation and MuQ, a pretrained feature-extraction model for music audio, to enhance generalisation abilities and extract meaningful audio features. Results show that the proposed A2S model achieves 4.98\% symbol error rate (SER) on the Quartets collection for classical music, which significantly outperforms the 15.3\% SER from the existing state-of-the-art \cite{alfaro-contrerasTransformer2024}. Additionally, our model achieves 20.92\% SER on the SheetSage-A2S dataset for popular music, serving as a strong benchmark for future research. The dataset, model, and code are made publicly available at: https://github.com/Multimodal-Music-Research-Lab/SheetSage2Kern_model.
Figures
Figures from the paper (3 more)
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.