REVIEW 5 major objections 6 minor 33 references
A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization
T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read SpectMamba, a Mamba-based network with note-f0 decoding and confidence binary regularization, claims top accuracy on three singing melody extraction benchmarks while running twice as fast.
desk verdict Plausible semi-supervised SME method with a useful combination of existing ideas, but the undisclosed MedleyDB split and missing semi-supervised baselines keep the headline claims from being substantiated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the note-f0 decoder, which realizes a coarse-to-fine prior: it expands the note prediction $\hat{P}_{\mathrm{note}}$ by replicating each note class across all f0 classes that belong to that note, applies softmax to form attention weights, and multiplies them onto the raw f0 prediction before a final MLP: $\hat{P}_{f0} = \mathrm{MLP}(\mathrm{Softmax}(\hat{P}_{\mathrm{note}}) \cdot P_{f0})$. This makes the f0 estimate concentrate inside the predicted note's frequency band. Around it sit the SpectMamba encoder, which patches the spectrogram and processes it with a bidirectional state-space model in both time directions, and the confidence binary regularization module, which uses an EMA-tracked global top-k confidence to split each unlabeled prediction into positive and negative parts and enforce weak-strong consistency with cross-entropy.
What would settle it
Retrain SpectMamba with the note replication in Eq. (8) replaced by a constant attention mask (no note information) while keeping everything else identical: if overall accuracy stays within measurement noise on ADC2004 and MIREX 05, the note-f0 decoder is not load-bearing. Alternatively, perturb the note-to-bin assignment by half a bin and measure the change in overall accuracy.
Extended reading notes
Core claim
The paper's central claim is that a vision-Mamba encoder, a note-f0 decoder, and a confidence binary regularization module jointly yield the highest singing melody extraction accuracy among the models compared. The note-f0 decoder predicts a note sequence and a frequency sequence in parallel, then uses the note prediction as an attention mask on the f0 prediction so that the model first commits to the musical note and then refines the fundamental frequency inside that note's frequency range. The confidence binary regularization module divides each unlabeled sample's predictions into a positive part (top-k confident classes) and a negative part (the rest), with the boundary set by an EMA-updated global confidence, and enforces consistency between weakly and strongly augmented versions. On ADC2004, MIREX 05, and MedleyDB the paper reports higher overall accuracy than FTANet, TONet, and S2Former, and it attributes the accuracy gain to the note supervision and the efficiency gain to the linear-complexity Mamba encoder.
Load-bearing premise
The note-f0 decoder assumes a fixed, known mapping from musical note classes to CFP frequency bins, and that replicating a note prediction across all bins belonging to that note is a valid attention prior; the paper does not specify this mapping.
Editorial extensions
If this is right
- Melody extraction at 2x inference speed and 1/10 memory use makes real-time or on-device singing melody extraction practical at the same accuracy level as transformer models.
- Note-level supervision gives a principled way to inject musical structure, so the approach could transfer to other pitch-tracking tasks where notes are defined.
- Confidence binary regularization lets unlabeled music contribute to training, reducing the annotation burden for melody extraction systems.
- The reported gains are consistent across three datasets, suggesting the method generalizes beyond a single corpus.
Reading between the lines
- If the fixed note-to-bin mapping is why the note-f0 decoder works, the decoder should degrade gracefully as the frequency resolution per note decreases; one testable extension is to train variants with 30 or 120 bins per octave and compare overall accuracy.
- The note-f0 decoder's masking idea could also serve as a differentiable pitch-range prior in source separation or instrument transcription, not just melody extraction.
- A stronger claim implicit in the ablation is that note supervision and CBR contribute additively; a direct test would be to remove both simultaneously and see whether the drops compound or interact.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SpectMamba, a Mamba-based network for semi-supervised singing melody extraction. The model uses CFP spectrogram patches, a bidirectional state-space encoder with linear complexity, a coarse-to-fine note-f0 decoder that uses note predictions to guide f0 prediction, and a confidence binary regularization (CBR) module that applies consistency supervision between weakly and strongly augmented unlabeled samples using an adaptive top-k confidence split. Experiments on ADC2004, MIREX 05, and MedleyDB report state-of-the-art results, with ablations showing contributions from the note-f0 decoder and CBR, and an efficiency claim of 2x inference speed and 1/10 GPU memory versus S2Former.
Significance. If the results hold, the paper makes a useful contribution to singing melody extraction: it is, to my knowledge, the first Mamba-based SME model, and the linear-complexity encoder directly addresses the quadratic cost of transformer baselines. The note-f0 coarse-to-fine idea is a plausible way to inject musical prior knowledge, and the CBR module is a principled extension of consistency regularization to the dense prediction setting. The paper includes an ablation study that directionally supports both novel components, and it provides a GitHub link to code. However, several specification gaps and evaluation weaknesses currently prevent the contributions from being fully verified.
major comments (5)
- [Section III-A / Table II(c)] Section III-A states that 35 MedleyDB tracks are used for training and 12 MedleyDB tracks are used for testing, but it never states that these two sets are disjoint or provides a split procedure. If any of the 12 test tracks are contained in the 35 training tracks, the results in Table II(c) (e.g., SpectMamba OA 72.62 vs. S2Former 71.35) would be invalid because the model would have already seen the test labels. Even if the split is actually disjoint, the missing disclosure prevents verification and reproduction of the experimental setup. The authors must explicitly state that the training and test subsets of MedleyDB are disjoint, provide the exact track lists or split indices, and if any overlap exists, recompute the MedleyDB results on a truly held-out set.
- [Section II-C / Eq. (8)] The note-f0 decoder relies on a fixed mapping from V note classes to F f0 bins, but the paper never specifies V, how note classes are defined, how the repetition operation in hat(P_note) assigns each f0 bin to a note class, or how Q_note is derived from Q_f0. With 60 bins per octave and a 31-1250 Hz range, a single note class may cover many bins whose frequency content varies; if the mapping is coarse or misaligned, Eq. (8) can suppress correct f0 classes and the reported gains from note supervision would not transfer to other settings. This mapping is required for reproducibility and to assess whether the note prior is actually valid. Please specify the mapping (e.g., a MIDI-note-to-CFP-bin table), the number of note classes V, and the exact procedure used to obtain Q_note from Q_f0.
- [Section II-D / Eq. (10)] The CBR algorithm is underspecified. Equation (10) uses k as a fixed number of positive classes in the computation of p_t, but the following sentence says 'adjusting k until the cumulative value exceeds the global top-k confidence,' which implies k is not fixed but is determined per sample. The EMA update for the global confidence mu_t is never written out; the paper only defines p_t and states that EMA is used. This is not a complete algorithm, and it is load-bearing because CBR is the central semi-supervised contribution. Please provide the explicit update rule for mu_t (including the decay schedule and initialization), define the positive-set selection rule unambiguously (whether k is a fixed integer or a per-sample variable), and report the values of k, mu_0, and the EMA decay hyperparameter.
- [Section III-D / Tables I and II] The evaluation uses only 12, 9, and 12 test tracks for ADC2004, MIREX 05, and MedleyDB, respectively, and reports single runs without standard deviations, confidence intervals, or significance tests. The OA improvements over S2Former are 2.24, 1.85, and 1.27 percentage points; with such small test sets, these differences may not be statistically distinguishable from noise. Please report per-track results, bootstrap confidence intervals, or paired significance tests, and if multiple runs are performed, report means and standard deviations. This is necessary to support the claim in Section III-D that the proposed model 'achieves the highest results in general.'
- [Section III-D / Table II] The paper claims semi-supervised singing melody extraction, but all three baselines (FTANet, TONet, S2Former) are supervised models; there is no semi-supervised baseline in the comparison. The contribution of CBR is evaluated only through an internal ablation (SpectMamba_B in Table I). To support the semi-supervised claim, the authors should compare against existing semi-supervised SME methods, such as the cited HKDSME [1] and MCSSME [3], trained under the same labeled/unlabeled data setup. Without such a comparison, it remains unclear whether the reported gains over S2Former come from the semi-supervised component or from the supervised architecture, and the claimed advantage in low-label regimes is not directly demonstrated.
minor comments (6)
- [Table I caption] The caption contains typos: 'SPECTMAMBAN STANDS FOR THE PROPOSED MODEL WITHOUT NOTE-F0 DECODER AND SPECTMAMBAB STANDS FOR THE PROPOSE MOEDEL WITHOUT CBR MODULE' should be corrected to 'SpectMamba_N stands for the proposed model without the note-f0 decoder and SpectMamba_B stands for the proposed model without the CBR module.'
- [Eqs. (11)-(12)] The notation CE(a_i^s, a_i^w) is ambiguous because both arguments are probability distributions; cross-entropy is normally defined between a predicted distribution and a target distribution or label. Please define the consistency loss precisely, e.g., as H(a_i^w, a_i^s) = -sum a_i^w log a_i^s, or state whether a symmetrized objective is used.
- [Eqs. (3)-(4)] The text says that for the backward direction the 1-D convolution and SSM are applied in reverse order, but Eq. (4) writes y_back = SSM_back(Conv_back(...)), which is the same order as the forward direction in Eq. (3). Please clarify the intended order or correct the equation.
- [Reference [17]] The title of reference [17] reads 'A scalable sarse Transformer model'; 'sarse' appears to be a typo for 'sparse'.
- [Figure 2] Figure 2 reports inference time and GPU memory usage, but the experimental conditions are not specified: no hardware, batch size, input audio length, number of averaged runs, or measurement methodology is given. Please provide these details so the 2x speed and 1/10 memory claims can be reproduced and interpreted.
- [Section III-A] The description of the unlabeled data is minimal: 1000 tracks from FMA are used, but it is not stated whether these are full tracks or excerpts, whether they are processed with the same CFP front-end, or how they are split into weak/strong augmentation pairs. A sentence on the preprocessing and usage of the unlabeled data would improve reproducibility.
Circularity Check
No circular derivation: the model's losses and evaluations are grounded in external labels and public benchmarks; self-citations are baselines, not load-bearing.
full rationale
The paper's derivation chain is self-contained. The supervised loss in Eq. (9) is standard cross-entropy against external f0 and note labels, and the note-f0 decoder in Eq. (8) uses predicted note probabilities as attention weights over f0 classes rather than as the target itself. The CBR loss in Eqs. (10)-(13) is a consistency regularizer between weakly and strongly augmented versions of unlabeled data, using the model's own predictions as pseudo-labels; this is conventional semi-supervised learning and does not inject the evaluation target into the fitted quantities. The headline results are measured on public datasets (ADC2004, MIREX 05, MedleyDB) with the mir_eval metrics, so the comparison is independent of the model's training targets. The baseline models (FTANet, TONet, S2Former) include prior work by the same research group, but they serve only as experimental comparators; no uniqueness theorem or prior result is invoked to force the proposed architecture. The two concerns raised—that the note-to-frequency-bin mapping in Section II-C is not specified and that the MedleyDB training/test split is not explicitly stated to be disjoint—are reproducibility and correctness risks rather than exhibited circular reductions. Without concrete evidence that the 12 MedleyDB test tracks overlap the 35 training tracks, possible data leakage cannot be scored as circularity under the requirement to quote a specific reduction, and the paper otherwise does not derive any central claim from quantities that already encode the target.
Assumptions & free parameters
free parameters (5)
- unlabeled loss weight omega =
0.1
- EMA decay rate for global confidence mu_t =
not reported
- top-k class count k =
not reported
- SpectMamba depth L and hidden dimensions =
not reported
- note-class count V and note-to-f0 bin mapping =
not reported
assumptions (4)
- domain assumption Vision Mamba bidirectional SSM features transfer from image patches to audio spectrogram patches
- domain assumption Note labels are consistently derivable from f0 annotations in the training data
- domain assumption Unlabeled FMA tracks contain vocal melodies relevant to singing melody extraction
- domain assumption Cross-entropy between weak and strong positive-negative distributions is a valid consistency objective
Cite this review
Pith. "Pith review of A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization." pith.science (2026). https://pith.science/paper/WKDZYFP4
@misc{pith2026250508681,
author = {Pith},
title = {Pith review of: A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization},
year = {2026},
howpublished = {\url{https://pith.science/paper/WKDZYFP4}},
note = {Machine review of arXiv:2505.08681}
}
read the original abstract
Singing melody extraction (SME) is a key task in the field of music information retrieval. However, existing methods are facing several limitations: firstly, prior models use transformers to capture the contextual dependencies, which requires quadratic computation resulting in low efficiency in the inference stage. Secondly, prior works typically rely on frequencysupervised methods to estimate the fundamental frequency (f0), which ignores that the musical performance is actually based on notes. Thirdly, transformers typically require large amounts of labeled data to achieve optimal performances, but the SME task lacks of sufficient annotated data. To address these issues, in this paper, we propose a mamba-based network, called SpectMamba, for semi-supervised singing melody extraction using confidence binary regularization. In particular, we begin by introducing vision mamba to achieve computational linear complexity. Then, we propose a novel note-f0 decoder that allows the model to better mimic the musical performance. Further, to alleviate the scarcity of the labeled data, we introduce a confidence binary regularization (CBR) module to leverage the unlabeled data by maximizing the probability of the correct classes. The proposed method is evaluated on several public datasets and the conducted experiments demonstrate the effectiveness of our proposed method.
Figures
Reference graph
Works this paper leans on
-
[1]
S Yu, X He, K Chen, and Y Yu, “HKDSME: Heterogeneous Knowl- edge Distillation for Semi-supervised Singing Melody Extraction Using Harmonic Supervision,” Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 545-553
work page 2024
-
[3]
S Yu, “MCSSME: Multi-Task Contrastive Learning for Semi-supervised Singing Melody Extraction from Polyphonic Music,” Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 365-373
work page 2024
-
[2]
RevNet: A Review Network with Group Aggregation Fusion for Singing Melody Extraction,
S Yu, X He, and Y Zhang, “RevNet: A Review Network with Group Aggregation Fusion for Singing Melody Extraction,” in Proc. ICME, 2024: 1-6
work page 2024
-
[4]
Audio cover song identification and similarity: background, approaches, evaluation, and beyond,
J Serra, E G ´omez, and P Herrera, “Audio cover song identification and similarity: background, approaches, evaluation, and beyond,” in Proc. Advances in Music Information Retrieval, 2010, Springer, 307–332
work page 2010
-
[5]
Frequency temporal attention network for singing melody extraction,
S Yu, X Sun, Y Yu, and W Li, “Frequency temporal attention network for singing melody extraction,” in Proc. ICASSP, 2021, pp. 251–255
work page 2021
-
[6]
Improving query-by-singing/humming by combining melody and lyric information,
C C Wang and J S R Jang, “Improving query-by-singing/humming by combining melody and lyric information,” IEEE/ACM Trans. Audio Speech Language Processing, vol. 23, no. 4, pp. 798–806, 2015
work page 2015
-
[7]
Rethinking Singing V oice Separation With Spectral-Temporal Transformer,
S Yu, C Li, F Deng, and X Wang, “Rethinking Singing V oice Separation With Spectral-Temporal Transformer,” in Proc. APSIPA ASC, IEEE, 2021: 884-889
work page 2021
-
[8]
Music retrieval and recommendation: A tutorial overview,
P Knees and M Schedl, “Music retrieval and recommendation: A tutorial overview,” in Proc. SIGIR, 2015, pp. 1133–1136
work page 2015
Show all 33 references
-
[9]
Vision mamba: Efficient visual representation learning with bidirectional state space model,
L Zhu, B Liao, Q Zhang, X Wang, W Liu, and X Wang, “Vision mamba: Efficient visual representation learning with bidirectional state space model,” arXiv preprint arXiv:2401.09417, 2024
2024 arXiv
-
[10]
On the improvement of singing voice separation for monaural recordings using the MIR-1K dataset,
C L Hsu and J S R Jang, “On the improvement of singing voice separation for monaural recordings using the MIR-1K dataset,” IEEE Trans. Speech Audio Process, vol. 18, no. 2, pp. 310–319, 2010
2010
-
[11]
Medleydb: A multitrack dataset for annotation-intensive MIR research,
R M Bittner, J Salamon, M Tierney, M Mauch, C Cannam, and J P Bello, “Medleydb: A multitrack dataset for annotation-intensive MIR research,” in Proc. ISMIR, 2014, pp. 155–160
2014
-
[12]
FMA: A Dataset for Music Analysis,
M Defferrard, K Benzi, P Vandergheynst, and X Bresson, “FMA: A Dataset for Music Analysis,” in Proc. 2017, ISMIR. 316–323
2017
-
[13]
A streamlined encoder/decoder architecture for melody extraction,
T H Hsieh, L Su, and Y H Yang, “A streamlined encoder/decoder architecture for melody extraction,” in Proc. ICASSP, 2019, pp. 156–160
2019
-
[14]
mir eval: A transparent implementation of common mir metrics,
C Raffel, B McFee, E J Humphrey, J Salamon, O Nieto, D Liang, D P Ellis, and C C Raffel, “mir eval: A transparent implementation of common mir metrics,” in Proc. ISMIR, 2014
2014
-
[15]
Melody extraction from polyphonic music signals: Approaches, applications, and chal- lenges,
J Salamon, E G ´omez, D P W Ellis, and G Richard, “Melody extraction from polyphonic music signals: Approaches, applications, and chal- lenges,” IEEE Signal Processing Magazine, vol. 31, no. 2, pp. 118–134, 2014
2014
-
[16]
SpecTNT: a time-frequency transformer for music audio,
W T Lu, J C Wang, M Won, K Choi, and X Song, “SpecTNT: a time-frequency transformer for music audio,” in Proc. ISMIR, 2021, pp. 396–403
2021
-
[17]
A scalable sarse Transformer model for singing melody extraction,
S Yu, J Liu, Y Yu, and W Li, “A scalable sarse Transformer model for singing melody extraction,” in Proc. ICASSP, 2024, pp. 1071-1075
2024
-
[18]
TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music,
K Chen, S Yu, C Wang, W Li, T Berg-Kirkpatrick, and Dubnov S, “TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music,” in Proc. ICASSP, 2022, 621–625
2022
-
[19]
HANME: hierarchical attention network for singing melody extraction,
S Yu, Y Yu, X Chen, and W Li, “HANME: hierarchical attention network for singing melody extraction,” IEEE Signal Processing Letters, 2021, 28: 1006-1010
2021
-
[20]
A multidilation and multi-resolution fully convolutional network for singing melody extraction,
P Gao, C Y You, and T S Chi, “A multidilation and multi-resolution fully convolutional network for singing melody extraction,” in Proc. ICASSP, 2020, 551–555
2020
-
[21]
Main melody extraction with source-filter NMF and CRNN,
D Basaran, S Essid, and G Peeters, “Main melody extraction with source-filter NMF and CRNN,” in Proc. ISMIR, 2018
2018
-
[22]
V ocal melody extraction using patch-based cnn,
L Su, “V ocal melody extraction using patch-based cnn,” in Proc. ICASSP, 2018, 371–375
2018
-
[23]
Hierarchical graph-based neural network for singing melody extraction,
S Yu, X Chen, and W Li, “Hierarchical graph-based neural network for singing melody extraction,” in Proc. ICASSP, 2022, 626–630
2022
-
[24]
Searching for activation functions,
P Ramachandran, B Zoph, and Q V Le, “Searching for activation functions,” arXiv preprint arXiv:1710.05941, 2017
2017 arXiv
-
[25]
Combining spectral and temporal representations for multipitch estimation of polyphonic music,
L Su and Y H Yang, “Combining spectral and temporal representations for multipitch estimation of polyphonic music,” J. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 2015, 23(10): 1600- 1612
2015
-
[26]
Exploiting Frequency, Period- icity and Harmonicity Using Advanced Time-Frequency Concentration Techniques for Multipitch Estimation of Choir and Symphony,
L Su, T Y Chuang, and Y H Yang, “Exploiting Frequency, Period- icity and Harmonicity Using Advanced Time-Frequency Concentration Techniques for Multipitch Estimation of Choir and Symphony,” in Proc. ISMIR, 2016: 393-399
2016
-
[27]
Between homomorphic signal processing and deep neural net- works: Constructing deep algorithms for polyphonic music transcrip- tion,
L Su, “Between homomorphic signal processing and deep neural net- works: Constructing deep algorithms for polyphonic music transcrip- tion,” in Proc. APSIPA ASC. IEEE, 2017: 884-891
2017
-
[28]
Automatic music transcription leveraging generalized cepstral features and deep learning,
Y T Wu, B Chen, and L Su, “Automatic music transcription leveraging generalized cepstral features and deep learning,” in Proc. ICASSP, 2018, 401-405
2018
-
[29]
Melody extraction on vocal segments using multicolumn deep neural networks,
S Kum, C Oh, and J Nam, “Melody extraction on vocal segments using multicolumn deep neural networks,” in Proc. ISMIR, 2016
2016
-
[30]
Deep Salience Representations for F0 Estimation in Polyphonic Music,
R M Bittner, B McFee, J Salamon, P Li, and J P Bello, “Deep Salience Representations for F0 Estimation in Polyphonic Music,” in Proc. ISMIR, 2017, 63–70
2017
-
[31]
CNN Based Two-stage Multiresolution End-to-end Model for Singing Melody Extraction,
M T Chen, B J Li, and T S Chi, “CNN Based Two-stage Multiresolution End-to-end Model for Singing Melody Extraction,” in Proc. ICASSP, 2019, 1005–1009
2019
-
[32]
A neural harmonic-aware network with gated attentive fusion for singing melody extraction,
S Yu, Y Yu, X Sun, and W Li, “A neural harmonic-aware network with gated attentive fusion for singing melody extraction,” Neurocomputing, 2023, 521: 160-171
2023
-
[33]
AllMatch: Exploiting All Unlabeled Data for Semi- Supervised Learning,
Z Wu and J Cui, “AllMatch: Exploiting All Unlabeled Data for Semi- Supervised Learning,” arXiv preprint arXiv:2406.15763, 2024
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.