{"id":"00565b09-b97c-49ad-9142-09fc0c2319e1","arxiv_id":"2602.19778","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage pseudo-labeling plus selective knowledge-distillation pipeline lets chord-recognition students match or beat a larger pre-trained teacher while improving rare-chord accuracy.","lead":"This paper trains chord-recognition models in two stages: first on pseudo-labels from an open-weight teacher over 1,000 hours of unlabeled audio, then on ground-truth labels with selective knowledge distillation as a regularizer. The resulting student is reported to beat the teacher and the supervised baseline, with the largest gains on rare chord types.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unlabeled-test overlap could invalidate the 'student surpasses teacher' claim; no deduplication is reported.","rationale":"The reader identified both teacher training provenance and unlabeled/test overlap as the weakest point. I agree and focus on the unlabeled/test overlap because it directly threatens the controlled comparison behind the headline claim. The teacher's test performance is a fixed baseline; the student is the only model that trains on unlabeled audio. If that audio contains test tracks, the student gains an unfair transductive advantage, making the reported margins (Root 83.03 vs. 81.95) potentially a leakage artifact rather than a transfer result. This concern is concrete and testable, unlike the more general 'single-run' concern which affects confidence but not the core validity. Since the paper could be salvaged by a deduplication check and re-run, CONDITIONAL remains the right verdict, so I recommend no change to the reader's verdict. The concrete test I propose would either confirm the clean-transfer interpretation or require revision of the central claim.","tokens_in":13688,"tokens_out":7663,"duration_ms":74346,"concrete_test":"Perform a fingerprint-based overlap check between all unlabeled tracks (FMA, MAESTRO, DALI) and the 120 labeled test tracks, e.g., using chromaprint audio fingerprints, exact file hashes, and metadata-based canonical IDs. If any overlap is found, rerun the entire two-stage pipeline with the overlapping unlabeled tracks removed and re-evaluate BTC (CL) full on the test set. If the Root metric drops below 82 (vs. teacher's 81.95) or any of the seven mir_eval metrics no longer exceeds the teacher, the paper must report deduplicated numbers and temper the 'surpasses teacher' claim. If no overlap is found, the concern is resolved and the claim stands as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (BTC student surpasses the teacher across all seven mir_eval metrics, §5.3 Table 3) assumes a controlled comparison: the teacher is a fixed baseline, and the student's Stage-1 training on >1,000 hours of unlabeled audio must not give the student access to the 120-song test set. The paper never verifies this. §4.1 lists FMA, MAESTRO, and DALI as unlabeled sources, and Isophonics, McGill, RWC, and USPop as labeled sources, but no deduplication step is reported. DALI is especially concerning: it consists of YouTube music videos and can contain the same commercial recordings used in the labeled test set (e.g., a Beatles track from Isophonics is available on YouTube). If a test track appears in the unlabeled pool, the Stage-1 student trains on its audio features via teacher pseudo-labels, gaining transductive knowledge of test inputs. The teacher, being frozen, never benefits. This alone could explain part of the 1.1–3.2 point margin by which the student surpasses the teacher, and it would invalidate the clean-transfer interpretation. The paper's conclusion that pseudo-label pretraining produces 'generalized representations' (§6) depends on this control being clean.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for automatic chord recognition (ACR). Stage 1 uses a pre-trained BTC model as a fixed teacher to pseudo-label more than 1,000 hours of unlabeled audio (FMA, MAESTRO, DALI) and trains student models (BTC and a new compact 2E1D) on those pseudo-labels. Stage 2 continually fine-tunes the student on ground-truth labels while applying a selective knowledge-distillation regularizer from the same teacher. Experiments on a 420/60/120 split of Isophonics/McGill/RWC/USPop report that, after Stage 2, the BTC student surpasses both the supervised-learning baseline and the teacher on all seven mir_eval metrics, with the largest gains on rare chord qualities such as Dim, Dim7, and Aug. The paper also provides an ablation showing KD protects against noisy-label fine-tuning.","tokens_in":13986,"tokens_out":4669,"duration_ms":46351,"significance":"If the reported results hold under controlled conditions, the paper makes a useful practical contribution: it shows that pseudo-label pretraining on large unlabeled corpora can reduce labeled-data requirements in ACR and improve rare-chord recognition, and that KD can serve as a forgetting regularizer during continual fine-tuning. The paper is explicit about using a fixed open-weight teacher, which is a realistic setting, and it evaluates with standard mir_eval metrics. The cross-architecture validation with 2E1D and the open-sourced web application are additional strengths. However, the empirical claims currently rest on a single split, a single training run, and an unverified separation between the unlabeled and test corpora, which puts the central 'student surpasses teacher' claim at risk.","major_comments":[{"comment":"No deduplication is reported between the unlabeled corpora (especially DALI) and the 120-song test set. DALI consists of YouTube audio for full-length commercial recordings, and the Isophonics/McGill/RWC/USPop test sets contain commercial pop/rock tracks, so a test track may well appear in the unlabeled pool. In Stage 1 the student is trained on teacher pseudo-labels for those exact test tracks, giving it transductive knowledge the frozen teacher never receives. This alone could explain part of the 1.1–3.2 point margin in Table 3. The paper must either demonstrate deduplication by artist/title metadata and/or audio fingerprinting, or retrain the pipeline after removing any overlapping tracks.","section":"§4.1, §5.3"},{"comment":"All numbers come from one 420/60/120 split and one training run. The key differences are small: BTC(CL) full root 83.03 vs teacher 81.95 and vs BTC(SL) 81.52, and several metrics differ by ~1–2 points. Without confidence intervals, multiple seeds, or statistical significance tests, these differences are within plausible run-to-run and split-to-split noise, especially with only 120 test songs. The central claim that the student 'surpasses' the teacher requires proper uncertainty quantification.","section":"§5.1, §5.3, Tables 1 and 3"},{"comment":"The comparison to Bortolozzo et al. [5] and Li et al. [7] is not controlled. The table reports metrics from the original papers evaluated on different test sets, while the text implies a direct comparison ('Prior work... results weaker'). Re-running these methods under the same test protocol, or clearly labeling the rows as 'as reported in the original paper, not directly comparable,' is necessary before drawing conclusions about pseudo-label quality.","section":"§5.3, Table 3"},{"comment":"The teacher is treated as a fixed external baseline, but its pre-training data are never stated. If the BTC teacher was trained on some of the labeled songs used here, the teacher's pseudo-labels on those songs (if they also appear in the unlabeled pool) would carry additional label information, further confounding the 'student surpasses teacher' claim. The authors should identify the teacher checkpoint and its training corpus, or explicitly state that this information is unavailable and explain how the interpretation changes.","section":"§4.2, §5.3"},{"comment":"The selective-KD thresholds θmin, θmax, K, the KD weight α, and temperature τ are all set empirically. The ablation in Table 2 varies only α and only under one noisy-label condition; the sensitivity of the final Stage-2 results to τ, θmin, θmax, and K is not reported. Since these hyperparameters directly control the regularizer that the paper claims is essential, the default choices need sensitivity analysis or a principled justification.","section":"§3.2.4, §4.2, Table 2"}],"minor_comments":[{"comment":"The text says 'We reserve 10% of the pseudo-labeled data for validation and 10% as a held-out test set,' but all reported metrics in Section 5 are evaluated on the 120-song labeled test set. The role of the pseudo-labeled held-out set needs clarification.","section":"§4.2"},{"comment":"For K=1, w(c) approaches 0 as c→1; with the chosen K=0.8 this is nonnegative, but the nonnegativity constraint is not stated. Add a brief constraint or remark.","section":"§3.2.4, Eq. (8)"},{"comment":"The caption contains an incomplete phrase 'Data C:maj' and is otherwise hard to parse. Please rewrite for clarity.","section":"Figure 3 caption"},{"comment":"The footnote 'We report only the metrics available in the original papers with our test set' is self-contradictory if the original papers used different test sets. Please state precisely how these numbers were obtained.","section":"Table 3 footnote"},{"comment":"Reference [19] is formatted incorrectly ('L. S. Helong Zhou'); the author list should match the original paper. Also, several arXiv references lack version/accession dates.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The deduplication issue is the most consequential. If any test track appears in the unlabeled pool, the central result could collapse. I would ask the authors for a clear statement of how DALI and other unlabeled data were checked against the labeled test set, and ideally a rerun after removing overlaps. The lack of repeated runs/confidence intervals is also a serious but fixable weakness. The paper's overall idea is solid and the experiments are extensive, so I do not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent empirical study. The new bits are the two-stage decoupling of pseudo-label pretraining from labeled fine-tuning, the selective-KD weighting, and the 2E1D architecture. The evaluation is internally consistent: the student trained on pseudo-labels alone reaches 99% of the teacher, and after fine-tuning with KD it edges past both the supervised baseline and the teacher on all seven mir_eval metrics, with big gains on rare chord qualities. Those are results worth reporting. The paper also ships a working app and model weights, which is more than many MIR papers do.\n\nThe main soft spot is the one the stress-test flags, and I think it lands. The central 'student surpasses teacher' claim requires the >1000 hours of unlabeled audio to be disjoint from the 120-song test set. The paper lists FMA, MAESTRO, and DALI as unlabeled sources, and DALI is drawn from YouTube music videos. It is entirely plausible that a Beatles track in the Isophonics test set also appears in the DALI pool. If so, the Stage-1 student sees test audio features via teacher pseudo-labels, while the frozen teacher never does. That transductive leakage alone could explain part of the 1.1–3.2 point margin. The paper reports no deduplication, and does not state what the teacher was trained on either, so we cannot rule out overlap there too. This is not a proof that the claim is wrong; it is a missing control that makes the headline result unverifiable as reported.\n\nThe other weaknesses are proportionate. All results are from a single training run on a 120-song test split with no confidence intervals or significance tests. The hyperparameters α, τ, and the selective-KD thresholds are chosen empirically on the same test set, which inflates apparent gains. The 'agreement with teacher' columns in Table 1 are partly circular since the teacher generates the pseudo-labels, but those columns are not the main evidence — the ground-truth metrics are — so this is minor.\n\nThe paper is clearly written, the method is easy to reproduce from the description, and the authors are honest about the teacher-quality limitation. It is not a breakthrough, but it is a useful practical contribution for MIR researchers who want to leverage unlabeled audio without building large labeled corpora. The serious referee should be sent in with one explicit request: verify disjointness between unlabeled and test sets (or quantify the overlap), and rerun at least the headline comparison a few times to show the margin is not noise. If that holds, the paper is solid. For now, I would treat the 'surpasses teacher' claim as conditional.\n\nI would send it to peer review rather than desk reject, but ask for the dedup check before acceptance.","headline":"A practical two-stage pseudo-labeling + KD pipeline for chord recognition with a plausible but under-supported claim that the student beats the teacher; the missing dedup check against the test set is the real worry.","tokens_in":14472,"tokens_out":1642,"would_cite":true,"duration_ms":17476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a chord-recognition student on pseudo-labels from a pre-trained teacher, then fine-tuning on ground truth with selective knowledge distillation, lets the student surpass both the teacher and a fully supervised","keywords":["automatic chord recognition","pseudo-labeling","knowledge distillation","semi-supervised learning","continual learning","rare chord quality","music information retrieval","selective knowledge distillation"],"falsifier":"Check whether any of the 120 test songs appear in the pseudo-label pool or in the teacher's training corpus. If the student's margin over the teacher on the seven metrics shrinks or inverts after removing such tracks from both the unlabeled pool and the teacher's evaluation, the leakage explanation is confirmed; if it persists on a disjoint subset, the transfer claim survives.","tokens_in":13574,"feed_emoji":"🎵","tokens_out":6867,"duration_ms":54927,"temperature":0.7,"pith_summary":"The paper sets out to show that automatic chord recognition does not need large amounts of hand-aligned labels if a good pre-trained model is available. To establish this, the authors build a two-stage pipeline: first, a pre-trained teacher (the BTC model) pseudo-labels over 1,000 hours of unlabeled audio, and a student is trained solely on those labels; second, the student is fine-tuned on ground-truth labels while selective knowledge distillation stabilizes the adaptation. The main result is that the best student surpasses both the fully supervised baseline and the teacher itself on all seven standard evaluation metrics, with the largest gains on rare chord qualities (overall average chord quality accuracy rises from 29.0% to 39.5%, while Dim7 recognition goes from 0% to 45.6%). The stakes: if this holds, chord recognition models can be trained on a fraction of the manually curated labels, and open-weight models can serve as practical teachers for specialized audio understanding tasks.","feed_headline":"Pseudo-label pipeline beats teacher on all 7 chord metrics","feed_subtitle":"Two-stage training on 1,000+ hours of unlabeled audio lifts rare-chord ACQA from 29% to 39.5%.","key_machinery":"The load-bearing mechanism is the two-stage pipeline with selective knowledge distillation. Stage 1 uses a pre-trained teacher (BTC) to produce frame-wise pseudo-labels (teacher argmax, no confidence filtering) for over 1,000 hours of diverse unlabeled audio, and trains a student on those labels until convergence; three pooled corpora are chosen so that the 12 chord roots are nearly uniform (98.4% uniformity), removing the need for pitch-shifting augmentation. Stage 2 initializes from the Stage-1 student and continues training on ground-truth labels while adding a KD loss to the cross-entropy, weighted by an asymmetric confidence function that discards low-confidence teacher predictions (bel","core_discovery":"On its own terms, the paper claims that a knowledge-distilled student can outperform its own teacher when the teacher is used only to generate pseudo-labels for unlabeled audio, and the result transfers across architectures. Stage 1 gives the BTC student over 99% of the teacher's score and a compact transformer student (2E1D) about 97%; Stage 2, with ground-truth fine-tuning and selective knowledge distillation, takes the BTC student to 83.03% root accuracy versus the teacher's 81.95% and the supervised baseline's 81.52%, and surpasses them on every one of the seven metrics. The most striking numbers are on rare chord qualities: Dim7 accuracy jumps from 0% (supervised) to 45.6%, and the clas","pith_inferences":["The recipe generalizes beyond chords: any MIR task with expensive labels (key estimation, beat tracking) could borrow a strong open-weight teacher, pseudo-label a large unlabeled pool, then fine-tune with selective KD; the paper hints at this but does not test it.","The 'student surpasses teacher' result is sensitive to the teacher's own training data: if the teacher was trained on songs that appear in the 120-song test set, part of the margin is leakage rather than transfer; a clean check is to rerun with a fully documented teacher.","Selective KD's asymmetric weighting implies that overconfident teacher predictions are often wrong for rare chords; this could be tested by ablating the overconfidence cap and measuring per-class ACQA.","Because the two stages are decoupled, the pipeline offers a practical timeline: deploy with pseudo-label training first, then incorporate human labels as they arrive, making it suitable for continual label acquisition in production systems."],"forward_implications":["Labeled-data requirements drop: with only 210 labeled songs (50% of the training set), the BTC student already beats the teacher on most metrics, and with the full 420 songs it beats the teacher on all seven.","Rare chords become realistic: chord qualities effectively absent from supervised training (Dim7 at 0%) reach 45.6% under the pipeline, and ACQA improves by 10.5 points.","KD as a regularizer transfers across architecture: the wider, purely transformer 2E1D student needs stronger KD (α=0.5) and still improves over its supervised baseline, while the deeper BTC peaks at α=0.3.","Noise robustness: when annotations are misaligned, without KD both models degrade severely (the wider model collapses), while with selective KD they recover most of the gap.","Natural root coverage replaces pitch shifting: pseudo-labels from diverse unlabeled corpora produce near-uniform chord-root coverage (98.4% uniformity), so the pipeline avoids pitch-shift artifacts and augmentation cost."],"fun_headline_variants":["Student beats teacher on all 7 chord metrics","Rare chord accuracy leaps from 0% to 45.6%","Pseudo-label pipeline beats teacher on chords","Two-stage training with KD tops supervised baseline"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"All headline improvements are measured against a teacher model whose training data is not disclosed; if the teacher saw the test songs, or if the 1,000+ hours of 'unlabeled' audio include the test tracks, the student's lead over the teacher could be an artifact of data leakage.","fun_headline_variants_meta":{"raw":{"variants":["Student beats teacher on all 7 chord metrics","Rare chord accuracy leaps from 0% to 45.6%","Pseudo-label pipeline beats teacher on chords","Two-stage training with KD tops supervised baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1378,"prompt_tokens":838,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":582,"tokens_out":540,"duration_ms":5144,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:31:15.408579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether any of the 120 test songs appear in the pseudo-label pool or in the teacher's training corpus. If the student's margin over the teacher on the seven metrics shrinks or inverts after removing such tracks from both the unlabeled pool and the teacher's evaluation, the leakage explanation is confirmed; if it persists on a disjoint subset, the transfer claim survives.","supporting_citations":[],"review_version":1}