{"id":"733b81c1-240a-46f5-841a-3c438055298f","arxiv_id":"2608.06633","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 46-hour frame-level pansori mode corpus with multi-representation classifiers shows mode-relevant generalization and representation disagreement patterns consistent with musicological theory.","lead":"Researchers built a 46-hour, expert-labeled dataset of traditional Korean pansori singing and tested four different audio representations for classifying musical modes frame by frame. The system keeps most of its accuracy when tested on completely unheard works, and the way different representations disagree maps onto how musicians actually describe the modes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The work-level split is under-specified: unless singer and recording-session identity are disjoint across folds, the 2.1–3.6 F1 gap does not establish mode generalization rather than timbre or session memorization.","rationale":"The reader's weakest assumption identifies exactly the same point: the Work-level Split is assumed to prevent repertoire memorization, but Section 3.3 does not establish that singers, accompanists, or studios are disjoint across folds. I agree with that assessment. The main quantitative result, that well-represented modes degrade by only 2.1–3.6 F1 when entire works are held out, is the strongest evidence for the paper's central claim, and it is only as strong as the split's ability to block shortcut learning. Since pansori mode is defined in part by vocal timbre, a model could plausibly learn singer-dependent timbral markers that transfer across batang, making the split gap look like mode generalization. The paper deserves credit for releasing the dataset and code, for using two complementary split strategies, and for the source-separation ablation on Changjo; none of these, however, supplies the missing singer/session disjointness check. The concern is addressable and does not by itself invalidate the dataset or the per-representation observations, so I would not move the verdict to reject. On the other hand, the paper's central generalization claim should not be accepted as fully established until the leakage check is run. The reader's conditional verdict is the right level of confidence, so I recommend no change.","tokens_in":10919,"tokens_out":5582,"duration_ms":54254,"concrete_test":"Using the released GitHub metadata, reconstruct the Work-level folds and compute singer and recording-session overlap between training and test. Then retrain Mel-Original and the ensemble on a strict singer-disjoint split (hold out all tracks by one or more singers and, if possible, all tracks from the same recording session) and compare masked macro-F1 and per-mode F1 to Table 2. If F1 on never-seen singers falls more than about 5 points below the reported Work-level numbers, especially for Ujo and Changjo, the reported small split gap is inflated by singer or session leakage. If the metadata shows zero singer and session overlap across folds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference in Section 5.2 is that the Work-level Split rules out memorization, so the small gap between Daemok-Shared and Work-level F1 shows the models learn mode-relevant features. This inference requires that the split separates not only melodic passages but also singer identity, accompaniment, and recording conditions. Section 3.3 describes partitioning each batang into two segments and assigning danga only to training, but it does not report whether singers, gosu players, or studios recur across folds. In a corpus of 396 tracks spanning only five batang, the same noted sorikkun and gosu routinely appear in multiple batang, and complete recording sessions can contain several batang. If a held-out batang segment shares a singer or session with training tracks, a model can use vocal-timbre or recording-environment shortcuts that transfer across batang; the 2.1–3.6 F1 gap is then compatible with shortcut learning. The claim that models generalize across singers and repertoire is thus load-bearing on a no-leakage condition that the manuscript does not verify. This is a correctness risk, not an accusation: the finding would remain true if the metadata shows singer-disjoint folds, but that evidence is absent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a 46-hour, frame-level pansori mode classification dataset spanning all five canonical batang, with seven expert-provided mode labels collapsed into four classes. It evaluates four input representations (mel spectrogram, F0 contour, MIDI piano roll, and a CultureMERT SSL embedding) plus a theory-grounded ensemble under two split strategies: Daemok-Shared, which tests generalization across singers on known passages, and Work-level, which holds out entire batang segments. The central claim is that the small F1 gap between the two splits for the three well-represented modes (2.1–3.6 points) shows the models learn mode-relevant features rather than memorizing repertoire, and that cross-modal disagreement patterns recover documented musicological distinctions, including Changjo's dependence on percussion and the failure of generic SSL pretraining on the Ujo–Gyemyeonjo distinction.","tokens_in":11142,"tokens_out":4847,"duration_ms":43702,"significance":"If the main generalization claim holds, this is a substantial contribution: the released frame-level annotation corpus is the first of its scale for pansori, and the multi-representation design is theoretically motivated by the multidimensional definition of jo. The paper also makes concrete, falsifiable claims about modality-specific cues (e.g., Changjo is identified primarily by drum accompaniment, and MIDI/F0 capture scale-theoretic versus micro-pitch dimensions) and provides publicly available code and annotations, which are clear strengths. The qualitative comparison with published score-based analyses of modern changjak pansori is a valuable interpretability check. However, the central quantitative conclusion depends on verification of the no-leakage property of the Work-level Split and on statistical stability of the reported F1 values, neither of which is currently demonstrated.","major_comments":[{"comment":"The Work-level Split is intended to rule out repertoire memorization, but the no-leakage condition is not verified. The corpus of 396 tracks spans only five batang, and the same sorikkun and gosu repeatedly appear across batang; recording sessions can also contain multiple batang. Section 3.3 describes partitioning each batang into two segments and assigning danga exclusively to training, but it does not report whether singer identity, accompanist identity, or recording-session/studio conditions are disjoint across folds. If a held-out segment shares a singer or session with training tracks, a model could exploit vocal-timbre or recording-environment shortcuts that transfer across batang, and the 2.1–3.6 point F1 gap reported in Section 5.2 would no longer establish mode-relevant generalization. Please add a metadata-based analysis of overlap across folds or otherwise demonstrate identity-disjoint splits.","section":"§3.3, Work-level Split"},{"comment":"The central quantitative claim is the small gap between Daemok-Shared and Work-level F1 scores, but every number in Table 2 appears to come from a single training run. No multiple seeds, confidence intervals, or significance tests are reported. Given typical CRNN variance on imbalanced frame-level tasks, a 2–4 point F1 gap can easily fall within run-to-run noise. Please report means and standard deviations over several seeds, or paired bootstrap intervals, for at least the main representations and both split conditions.","section":"§5.2, Table 2"},{"comment":"The results are not contextualized against simple baselines. The class distribution is heavily imbalanced (Gyemyeonjo 58.2%, Ujo 19.0%, Aniri 15.2%, Changjo 1.8% of labeled frames), and the reported masked macro-F1 values (roughly 0.54–0.87) are difficult to interpret without a majority-class, class-prior, or frequency-based baseline. Without such baselines, the absolute accuracy levels and the relative ranking of representations cannot be properly assessed. Please add at least a majority-class and a class-prior baseline to Table 2.","section":"§5.2, Table 2"},{"comment":"All frame-level labels were produced by a single expert annotator, and the qualitative validation in Section 6 appears to involve the same co-author with formal pansori training. This raises a circularity concern for the claim that cross-modal disagreement 'recovers musicologically documented phenomena': the annotation and its interpretation share the same expert judgment. Please state explicitly whether the Section 6 reviewer is the annotator, and provide inter-annotator agreement on a subset of frames or a documented independent second-opinion protocol.","section":"§3.1 and §6"}],"minor_comments":[{"comment":"The abbreviation 'Gmj' is not defined in the caption or text; please spell out 'Gyemyeonjo' in the table header for readability.","section":"Table 2 caption"},{"comment":"Section 3.3 says '18 specific daemok' while Section 5.1 refers to a 'shared test set of 18 pieces'; please clarify whether these are the same items and describe how validation and test pieces are drawn in the Daemok-Shared split.","section":"§3.3 and §5.1"},{"comment":"The percentages in the left panel do not sum to exactly 100.0 because of rounding; please either adjust the values or note that percentages are rounded.","section":"Table 1"},{"comment":"The sentence 'the split gap averages only 2.1–3.6 points of F1 for all four signal-derived representations' is ambiguous because Table 2 shows different gaps per mode; please specify that this is the range across the three well-represented modes.","section":"§5.2"},{"comment":"Patterns A and B are described qualitatively; adding counts of how often each pattern occurs among the reviewed test frames would strengthen the claim of bidirectionality.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is in scope for ISMIR and the dataset release is potentially valuable. The requested revisions—leakage analysis, variance estimates, and baselines—are all empirically addressable within the paper's scope, so I would not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth your time. The real contribution is the 46-hour, frame-level pansori mode corpus, annotated by a professional across all five surviving batang, plus a diagnostic two-split protocol (Daemok-Shared vs. Work-level) that is a good idea for any small-repertoire tradition. The systematic comparison of mel spectrogram, F0, MIDI piano roll, and CultureMERT embeddings is appropriate given that pansori mode is defined jointly by scale, ornament, and timbre. The authors also deserve credit for framing the ensemble as an analytical instrument rather than a higher-accuracy system, and for the honest per-class analysis, especially the Changjo/source-separation result.\n\nThe central quantitative claim—that small split gaps (2.1–3.6 F1) on the well-represented modes indicate learning mode-relevant features rather than memorizing repertoire—is plausible but not fully established. The work-level split prevents melodic-passage overlap by design, but the paper does not report whether singers, accompanists, or recording sessions overlap across folds. In a corpus of 396 tracks drawn from only five batang, that overlap is likely. If train and test share a singer or a session, the model could use vocal-timbre or recording-environment shortcuts that transfer across batang, and the split gap would not mean what the abstract says. This is an addressable issue: a metadata table listing singer and session IDs per fold would settle it. As written, it is a real soft spot.\n\nSmaller issues: there are no error bars or multiple seeds, no majority-class baseline, and the annotation comes from a single expert without inter-annotator reliability. These are standard omissions for a first dataset paper, not fatal. One more thing: the qualitative validation is partly performed by the same co-author circle, and one of the \"published\" analyses used for comparison is by a co-author (ref [32]). That does not invalidate the agreement, but the paper should state the relationship so readers can judge independence.\n\nOverall, this is a solid dataset-and-benchmark paper that deserves a serious referee. I would recommend conditional acceptance: require the metadata check on the work-level split, add a majority baseline and some variance estimate, and clarify annotator independence. My own reading-group vote is yes, and I would cite the corpus if I worked in MIR or Korean musicology.","headline":"A genuinely useful new corpus and a sensible multi-representation benchmark for pansori mode, but the headline generalization claim rests on an unverified no-leakage condition in the work-level split.","tokens_in":11649,"tokens_out":2259,"would_cite":true,"duration_ms":22348,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frame-level pansori mode classification generalizes to unseen works, with F1 dropping only 2.1–3.6 points when entire batang are held out.","keywords":["pansori","jo mode classification","frame-level annotation","multi-representation audio","music information retrieval","microtonal ornament","work-level split","source separation"],"falsifier":"Re-run the Work-level Split with folds that are explicitly singer-disjoint and studio-disjoint, or probe whether the trained models can name the held-out batang via a trivial auxiliary classifier; if held-out performance drops by substantially more than the reported 2.1–3.6 F1 points, the small split gap was not evidence of mode generalization.","tokens_in":10725,"feed_emoji":"🎶","tokens_out":5096,"duration_ms":44933,"temperature":0.7,"pith_summary":"This paper tries to establish that a machine can label pansori mode (jo) frame by frame, and that it does so from the musical properties that define the mode rather than from memorized repertoire. Across the three well-represented classes—Ujo, Gyemyeonjo, and Aniri—performance drops only 2.1–3.6 points of F1 when entire batang are held out of training, which the authors read as evidence of genuine mode learning. The paper further claims that different input representations each capture a different theoretical dimension of mode: MIDI carries scale, F0 carries microtonal ornament, and the mel spectrogram carries vocal timbre. A reader should care because it offers a computational handle on a performance tradition where mode is intentionally multidimensional, and where scale alone cannot explain classification.","feed_headline":"Pansori modes survive whole-work holdouts within 3.6 F1","feed_subtitle":"A 46-hour expert-labeled corpus shows Ujo, Gyemyeonjo, and Aniri are learned from musical cues, not memorized songs.","key_machinery":"The machinery is a multi-representation frame classifier plus a two-way split diagnostic. Four inputs encode different facets of mode—mel spectrogram for timbre and envelope, F0 contour for vibrato and bent notes, MIDI piano roll for the scale skeleton, and a multi-culturally pretrained encoder for generic acoustic priors—all feeding convolutional-recurrent heads that emit per-frame logits over four classes. The diagnostic is the comparison between a Daemok-Shared Split, where the same passage appears in train and test with different singers, and a Work-level Split, where entire daemok are held out; the small gap between the two is the paper's operational test for whether the model learned mode properties instead of memorizing repertoire.","core_discovery":"On a new expert-annotated 46-hour corpus spanning all five surviving batang, four audio representations trained as frame-level classifiers stay within 2.1–3.6 F1 points of their performance on known daemok when entire works are held out, for Ujo, Gyemyeonjo, and Aniri. The paper argues this small split gap shows the models rely on mode-relevant acoustic features rather than song-specific shortcuts, with symbolic pitch and F0 representations generalizing slightly better than spectral ones. It also reports that source separation removes exactly the percussion cue that distinguishes Changjo, and that a multi-culturally pretrained SSL model fails specifically on the Ujo–Gyemyeonjo distinction, suggesting that generic pre-training captures less of the mode's performative dimension. The central discovery is thus not a single architecture but evidence that frame-level mode classification can be learned from complementary representations while remaining interpretable: cross-modal disagreement recovers musicologically documented cases where scale, ornament, and contour pull in opposite directions.","pith_inferences":["A natural test of the paper's split claim would be to build singer-disjoint and studio-disjoint folds; if F1 gaps grow beyond the reported 2.1–3.6 points, the generalization conclusion would need restating as singer familiarity rather than mode learning.","Because each representation specializes in a different dimension of mode, the disagreement patterns could be turned into a weak supervision signal for musicological annotation, flagging frames where human experts should re-listen.","The SSL model's failure on Ujo–Gyemyeonjo suggests that multi-cultural pre-training may encode scale statistics but miss culture-specific ornament; fine-tuning on microtonal contours rather than only the final adapter layer might close that gap."],"forward_implications":["If the split gap reflects genuine mode learning, frame-level pansori classification can be applied to unseen works, including modern changjak pansori, with the reported F1 levels.","Source separation is not harmless for this task: removing percussion erases the cue that defines Changjo, so practical annotation tools should retain the full mix.","MIDI and F0 representations are the most portable across repertoire, while spectral and SSL representations are more sensitive to the performative surface; any deployed system should combine symbolic and continuous inputs.","Cross-modal disagreement can be read musicologically: when an ensemble of representations contradicts the ground-truth label, the disagreement identifies passages where scale, ornament, and timbre conflict.","An ensemble is useful as an analytical instrument rather than as a top scorer; the paper's reported ensemble never beats the best single representation on overall masked macro-F1."],"supporting_citations":[{"why":"Supplies the musicological definitions and grouping of Gyemyeonjo, Ujo, Pyeongjo, Seollongje, and Gyeongdeureum that the class scheme relies on.","marker":"[8]"},{"why":"Inspires the two split strategies used to separate passage familiarity from mode generalization.","marker":"[10]"},{"why":"The source separator whose removal of percussion is shown to erase the Changjo cue.","marker":"[17]"},{"why":"The singing-voice transcription model that converts audio to the MIDI piano-roll input.","marker":"[21]"},{"why":"The pitch estimator that produces the F0 contour input capturing microtonal ornament.","marker":"[24]"},{"why":"The self-supervised music representation that anchors the SSL encoder used in the paper.","marker":"[26]"},{"why":"The multi-culturally pretrained encoder whose Ujo–Gyemyeonjo failure is a central diagnostic result.","marker":"[28]"},{"why":"Provides score-based mode analysis of a modern changjak work used for qualitative verification of the ensemble's transitions.","marker":"[32]"},{"why":"Provides the second score-based analysis of a modern changjak work that the ensemble's predictions reproduce.","marker":"[33]"}],"fun_headline_variants":["Pansori modes generalize within 3.6 F1 on held-out works","Source separation strips the percussion cue for Changjo","Generic SSL fails on the Ujo-Gyemyeonjo distinction","Cross-modal analysis recovers documented pansori phenomena","Whole-work holdout: pansori learning isn't just memorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's case for genuine mode learning rests on the assumption that the Work-level Split prevents repertoire memorization: that held-out batang segments share no melodic passages, singers, or recording conditions with the training folds, so a small train–test gap cannot come from leakage.","fun_headline_variants_meta":{"raw":{"variants":["Pansori modes generalize within 3.6 F1 on held-out works","Source separation strips the percussion cue for Changjo","Generic SSL fails on the Ujo-Gyemyeonjo distinction","Cross-modal analysis recovers documented pansori phenomena","Whole-work holdout: pansori learning isn't just memorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1907,"prompt_tokens":950,"completion_tokens":957,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":870}},"tokens_in":566,"tokens_out":957,"duration_ms":8735,"temperature":1.0,"reasoning_tokens":870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:10:35.970782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Work-level Split with folds that are explicitly singer-disjoint and studio-disjoint, or probe whether the trained models can name the held-out batang via a trivial auxiliary classifier; if held-out performance drops by substantially more than the reported 2.1–3.6 F1 points, the small split gap was not evidence of mode generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the musicological definitions and grouping of Gyemyeonjo, Ujo, Pyeongjo, Seollongje, and Gyeongdeureum that the class scheme relies on."},{"cited_title":"Raga classification from vocal performances using multimodal analysis","cited_arxiv_id":null,"evidence_quote":"Inspires the two split strategies used to separate passage familiarity from mode generalization."},{"cited_title":"Study on a few basic terms for building a ‘pansori-science’: Focusing on a comparative study of similar terms (『판소리학』정립을위한몇가지기본 용어고찰:유사용어의비교연구를중심으로 ),","cited_arxiv_id":null,"evidence_quote":"The source separator whose removal of percussion is shown to erase the Changjo cue."},{"cited_title":"Music artist classification with convolutional recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"The singing-voice transcription model that converts audio to the MIDI piano-roll input."},{"cited_title":"Pesto: Real-time pitch es- timation with self-supervised transposition-equivariant objective,","cited_arxiv_id":null,"evidence_quote":"The pitch estimator that produces the F0 contour input capturing microtonal ornament."},{"cited_title":"Large-Scale MIDI-based Composer Classification","cited_arxiv_id":"2010.14805","evidence_quote":"The multi-culturally pretrained encoder whose Ujo–Gyemyeonjo failure is a central diagnostic result."},{"cited_title":"Computational Analysis of Yaredawi YeZema Silt in Ethiopian Orthodox Tewahedo Church Chants","cited_arxiv_id":"2412.18788","evidence_quote":"Provides score-based mode analysis of a modern changjak work used for qualitative verification of the ensemble's transitions."},{"cited_title":"A comparative study of pak tong-sil’s and jeong cheol-ho’s <yi jun yeolsa-ga> (박동실과정철호 의 <이준열사가> 비교연구),","cited_arxiv_id":null,"evidence_quote":"Provides the second score-based analysis of a modern changjak work that the ensemble's predictions reproduce."}],"review_version":1}