{"id":"b9462971-2ebe-44f3-883b-a47c4a5516d6","arxiv_id":"2505.04419","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new ROD dataset and an ED-TCN model with don't-care chunking reportedly detect six Hindustani vocal ornaments with F1 near 90 on the in-domain split.","lead":"Researchers built a labeled dataset of about 4 hours of Hindustani vocal music marking six ornament types and trained a deep temporal model to detect them. The work aims to support music tutoring, singer identification, and expressive singing synthesis, though its benchmark numbers need scrutiny because the test pipeline may use ground-truth boundaries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1 scores rely on test-time chunking that consumes ground-truth event boundaries; no label-free inference procedure is described, so the central generalization claim is not yet supported.","rationale":"The reader's weakest assumption identifies the same core problem: the evaluation protocol requires ground-truth event boundaries to construct test chunks, and no label-free inference procedure is described. This is load-bearing because the paper's central claim is not merely that ED-TCN beats a CRNN on a fixed benchmark, but that it generalizes to real-world concert audio. If test-time chunking uses oracle event boundaries, the reported numbers cannot be reproduced by a deployed system, and the magnitude of the improvement over the baseline may be an artifact of the segmentation protocol rather than the model. The paper does provide useful resources: a new curated dataset, an annotation tool, and a clearly specified training-time strategy with ablations. The 'don't care' loss formulation in Eq. 3 is reasonable for training. However, the manuscript stops short of specifying how the model would be applied to unlabelled audio, which is a necessary component of any recognition system. The proposed concrete test would settle whether the concern lands: if fixed-window test performance is materially worse, the central generalization claim should be revised or the method extended with an inference-time segmentation strategy. Until that evidence is provided, rejecting the central empirical claim is appropriate, although the dataset contribution could stand alone if released cleanly.","tokens_in":14722,"tokens_out":7171,"duration_ms":72901,"concrete_test":"Re-run Experiment 1 and the Prasar Bharati experiment using a fixed 10-second window at test time, without access to ground-truth onsets or offsets, keeping the same trained models and training-time chunking. Compare F1 with the reported values; if the margin over the baseline shrinks or reverses, the event-aware chunking is responsible for the reported gains. Also inspect the released code or data split files to determine whether the Experiment 1 split is file-level or chunk-level; if chunk-level, repeat with file-level splitting to rule out leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section V-A, chunks are defined using the labeled event list: if an event spans a chunk boundary, the next chunk starts at that event's onset and the truncated tail is marked 'don't care'. Section VI states that all audio is segmented into 10-second chunks with this technique, including the test splits used for Tables III and IV. At inference on unlabeled audio, event onsets and offsets are unknown, so this chunking rule cannot be applied. The paper never specifies a label-free inference procedure (fixed windows, sliding windows, or a two-stage detector). Therefore the reported F1 scores (90.06/91.98 on ROD, 67.59 on Prasar Bharati) are measured under an oracle segmentation that uses ground-truth boundaries. This is precisely the scenario the paper's motivation says is harmful. The relative comparison to the CRNN baseline may still be internally consistent because the baseline is chunked the same way, but the absolute claim of generalization to real-world concert audio is not demonstrated. A second ambiguity is whether the 'random 70/20/10 split' of Experiment 1 is file-level or chunk-level; if chunk-level, overlapping chunks could leak the same audio content into both training and test sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ROD, a new dataset of expert-annotated vocal ornaments in Hindustani classical music, and proposes an ED-TCN model with don't-care labeling, periodic padding, and dilated convolutions for frame-wise ornament detection. The authors report F1 improvements over a CRNN baseline across several train-test configurations (Table III) and on a separate Prasar Bharati concert dataset (Table IV). The central claim is that the proposed model outperforms the baseline and generalizes across singers, ragas, and real-world concert audio.","tokens_in":14964,"tokens_out":5921,"duration_ms":59864,"significance":"The ROD dataset is a potentially valuable community resource: it contains 4.08 hours of expert-annotated recordings with six ornament classes, reports inter-annotator agreement (Cohen's Kappa 0.79), and the authors state that code, models, and demos will be released. The ablation study isolating the contributions of don't-care labeling, periodic padding, dilated convolutions, and chroma resolution is informative. However, the evaluation protocol has a load-bearing flaw: test-time chunking uses ground-truth event boundaries, so the reported generalization scores are not currently supported. If the authors provide a label-free inference protocol and re-run the experiments with proper splits and a fairly trained baseline, the paper could make a substantial contribution.","major_comments":[{"comment":"The chunking procedure defined in Section V-A uses ground-truth event onsets and offsets (o_ij, f_ij) to decide chunk starts and to assign 'don't care' labels to truncated tails. Section VI states that all audio is segmented into 10-second chunks using this technique, which necessarily includes the test splits used in Tables III and IV. At inference time on unlabeled audio, event onsets and offsets are unknown, and the paper does not describe any label-free inference protocol (fixed windows, sliding windows, or a separate event detector). Consequently, the reported F1 scores are produced under an oracle segmentation that consumes target labels. This invalidates the claims of generalization in Sections VII-A and VII-B. The authors must specify how test audio is chunked during inference and re-evaluate with a protocol that does not use ground-truth boundaries. In addition, the treatment of 'don't care' frames during test evaluation is unspecified; if truncated tails are marked as 'don't care' in test chunks, the paper must state whether those frames are excluded from the precision/recall computation.","section":"V-A and VI"},{"comment":"Experiment 1 is described as a 'random 70/20/10 train-test-validation split,' but the paper does not state whether the split is at the file level or the chunk level. The chunking technique in Section V-A produces overlapping chunks when an event crosses a boundary, so a chunk-level split can place overlapping chunks from the same audio file into both training and test sets, causing information leakage and inflating the reported F1. This ambiguity must be resolved; if the split was chunk-level, the experiments should be rerun with file-level splits.","section":"VII-A1 and Table II"},{"comment":"The baseline CRNN is trained for 100 epochs, while the proposed ED-TCN is trained for 3000 epochs. This 30-fold difference in training budget means the reported superiority in Table III could be attributed to under-training the baseline rather than to the architectural benefits of the proposed model. The authors should train the baseline to convergence (e.g., with early stopping or a comparable number of epochs) and report learning curves or a convergence criterion for both models.","section":"VI-A and VI-B"},{"comment":"The comparison on the Prasar Bharati dataset is not apples-to-apples: the 'Proposed Model (with FT)' is pre-trained on ROD and then fine-tuned, whereas the baseline is trained from scratch on the Prasar Bharati data. Any advantage may reflect the benefit of pre-training rather than the model design. The authors should compare against a baseline that is also pre-trained on ROD and fine-tuned, or report the proposed model trained from scratch on Prasar Bharati, to isolate the architectural contribution.","section":"VII-B and Table IV"}],"minor_comments":[{"comment":"In the periodic padding description, 'Xbottom = X[:,p]' appears to be a typo; it should likely be 'Xbottom = X[:p]' (first p rows). Please correct the notation.","section":"V-A, equations"},{"comment":"The paper states that 'Precision, Recall, and F1-score averaged over the classes' are reported, but it does not specify whether the averaging is macro (unweighted per class) or micro (weighted by support). This should be stated for reproducibility.","section":"VI-C"},{"comment":"The abbreviations 'Str1' and 'Ste1' are not defined in the caption or text; please define them as 'Training subset of Singer 1' and 'Testing subset of Singer 1' explicitly.","section":"Table II caption"},{"comment":"The text claims that don't care labelling 'consistently boosts F1 scores by 7–8%', but Table V shows an increase from 79.02 (without†) to 90.06 (with†), which is about 11 points. The text also says 'provides an increased performance of 11%' a few sentences earlier. Please reconcile these numbers.","section":"VII-C"},{"comment":"The GitHub URL in the abstract and introduction is given as 'https://github.com/madhavlab/2024 ornamentation' with a space; this will render as an invalid link. Please provide the correct URL.","section":"Abstract and I"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern identified by the reader is real and central: Section V-A's chunking consumes ground-truth event boundaries, and Section VI applies it to test audio. This invalidates the reported F1 scores as evidence for generalization. However, I do not think this is necessarily beyond repair: a fixed-window or sliding-window inference protocol could be described and evaluated, and the splits and baseline training can be clarified and improved. Given that the dataset and model are potentially useful and the code is promised to be released, I recommend major revision rather than outright rejection, contingent on a complete re-evaluation under a label-free protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The ROD dataset is the real contribution here. It fills a clear gap: expert-annotated Hindustani vocal recordings with six ornament types, documented annotation rules, and a reported inter-annotator agreement of 0.79. The human-in-the-loop annotation tool is also a practical step forward. I can see this dataset becoming a reference benchmark for ornament detection in Indian art music.\n\nThe model work is less convincing. The ED-TCN is borrowed, with periodic padding and dilated convolutions added, and the don't-care labeling is a sensible adaptation. But the evaluation has a load-bearing flaw. Section V-A defines chunk boundaries using labeled event onsets and offsets; Section VI says all audio, including test splits, is chunked with this technique. At inference on unlabeled audio, those boundaries are unknown, and the paper never describes a label-free procedure. So the F1 scores in Tables III and IV are measured under an oracle segmentation. The relative comparison against the CRNN baseline may be internally consistent, since both systems get the same helpful chunking, but the absolute claim that the model detects ornaments in continuous concert audio is not demonstrated. The stress-test note is on target.\n\nThere are other ambiguities. The random split granularity is not stated; if chunks are split randomly, overlapping chunks could leak content between train and test. The don't-care labels also remove hard boundary frames from training and possibly from evaluation, which further inflates the numbers. And the training budgets are lopsided: the baseline gets 100 epochs, the proposed model 3000.\n\nWhat the paper does well is the ablation study. Showing that don't-care labeling matters more than periodic padding or dilated convolutions is informative, and the qualitative analysis of ornament confusion (Kan vs Mind, etc.) is thoughtful.\n\nAs submitted, I would not accept the paper's central empirical claims. But the dataset deserves an outlet, and the evaluation problem is fixable. A revision that adds a label-free inference protocol and reports those numbers, while specifying split granularity and matching training budgets, would be worth publishing. I would send it to peer review rather than desk-reject, because the dataset alone justifies referee time.","headline":"The ROD dataset is a genuinely useful resource, but the evaluation's test-time chunking consumes ground-truth boundaries, so the reported F1 scores are not a valid measure of detection in continuous audio.","tokens_in":15474,"tokens_out":2918,"would_cite":true,"duration_ms":32479,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ornament detection in Indian classical vocals hits 90% F1","keywords":["ornamentation detection","Indian art music","Hindustani classical music","audio event detection","temporal convolutional network","active annotation","chromagram","don't care labelling"],"falsifier":"Take the trained model and score it on the ROD test files cut into fixed 10-second windows with no access to event boundaries, then compare $F_1$ to the event-aware-chunking result; if performance falls toward the baseline, the reported gains are conditional on knowing where ornaments start.","tokens_in":14534,"feed_emoji":"🎵","tokens_out":7251,"duration_ms":70415,"temperature":0.7,"pith_summary":"This paper works to make microtonal vocal ornaments—Kan, Mind, Murki, Nyas, Andolan, and Gamak—machine-recognizable in Hindustani classical vocal music. It introduces the ROD dataset, 4.08 hours of expert-sung studio recordings across four ragas, labelled through a two-stage active-annotation process with a reported inter-annotator agreement of 0.79. The core claim is that ornament detection succeeds when ornaments are treated as events whose full temporal shape is preserved: audio is chunked so that an ornament crossing a boundary is never cut in both chunks, and the cut portion is masked with a \"don't care\" label during training. On the ROD general split the proposed ED-TCN model reaches 90.06 $F_1$ without a collar and 91.98 with a 200 ms collar, against 74.35 for the CRNN baseline, and after fine-tuning on real-world concert recordings it reaches 67.59 $F_1$ versus 53.68 for a baseline trained from scratch. A sympathetic reader would care because ornamentation is central to Indian art music, not an optional decoration, and this is a working recipe for detecting it automatically.","feed_headline":"Ornament detection in Indian classical vocals hits 90% F1","feed_subtitle":"Event-aware chunking and don't-care labels beat a CRNN baseline and transfer to concerts and singers.","key_machinery":"The load-bearing mechanism is the interaction between event-aware chunking and masked training. Each 10-second chunk starts at the onset of any ornament that would otherwise be cut by the boundary, and the already-partially-seen tail of a truncated ornament is labelled $\\dagger$ (\"don't care\"); the loss in Eq. (3) then ignores those frames. This is paired with an encoder-decoder temporal convolutional network (ED-TCN), where the encoder applies dilated 1D convolutions with rates 1, 2, 3, 4 and max-pooling, the decoder upsamples symmetrically, and every convolution uses periodic padding that wraps the chroma rows (pitch classes) instead of zero-filling. Periodicity in pitch and dilated receptive fields let the model see a full ornament's temporal structure, while the $\\dagger$ mask stops fragmented labels from teaching the model to misclassify partial events.","core_discovery":"On its own terms, the paper establishes a working recipe for automatic recognition of vocal ornaments in Hindustani classical music. It contributes the ROD dataset, 212 studio recordings (4.08 hours) by two expert singers across four ragas, strongly labelled for six ornament types with a two-stage active-annotation protocol and a reported Cohen's kappa of 0.79. Its proposed ED-TCN model takes 120-bin chromagrams as input and adds three adaptations to a standard temporal-convolution backbone: \"don't care\" labels that mask truncated ornament fragments during loss computation, periodic padding that wraps pitch classes instead of zero-padding, and dilated convolutions that enlarge the temporal context. Across general, intra-singer, inter-singer, and raga-specific splits, the model reports $F_1$ between 76.72 and 92.55 without a collar and consistently beats the CRNN baseline; on a separate real-world concert dataset, fine-tuning lifts $F_1$ to 67.59 against 53.68 for a from-scratch baseline. The ablation study identifies \"don't care\" labelling as the largest single contributor, worth 7–8 points of $F_1$ over the same model without it.","pith_inferences":["Inference beyond the paper: the \"don't care\" masking idea is transportable to any event-detection task whose events are identified by their internal temporal shape—animal vocalizations, drum strokes, coughs—where fixed-window cropping corrupts the evidence.","Inference beyond the paper: a field-ready system would need a label-free way to choose chunk starts on long audio, since the paper's chunking procedure uses ground-truth onsets; a natural extension is a boundary-proposal network that predicts event starts and then feeds those chunks to the detector.","Inference beyond the paper: the same two-stage active-annotation workflow could produce comparable ornament datasets for other oral music traditions, with the event label scheme adapted to each tradition's ornament inventory.","Inference beyond the paper: the confusion patterns (Kan–Mind, Murki–Kan, Andolan–Mind) suggest that adding raga or gharana metadata as conditioning input could break the structurally ambiguous cases the current model still merges."],"forward_implications":["At 90.06 $F_1$ on the general ROD split (91.98 with a 200 ms collar), the model is a usable front end for music-pedagogy feedback, singer-identification features, or expressive-synthesis conditioning, tasks the paper lists as downstream motivations.","Inter-singer experiments keep $F_1$ above 80 without a collar when training on one singer and testing on the other, implying the detector learns ornament structure rather than singer-specific timbre.","Raga-specific experiments stay near or above 75 $F_1$, and the concert-recording experiment reaches 67.59 after fine-tuning, so the approach extends beyond studio lessons to broadcast-quality archival audio after vocal-stem separation.","Because \"don't care\" labelling contributes more than any other component, the paper's recipe implies that future ornament datasets should be chunked with event boundaries in mind from the start, not with fixed windows."],"supporting_citations":[{"why":"Supplies the CRNN baseline and the frame-wise singing-technique detection formulation this work compares against.","marker":"[3]"},{"why":"Provides the encoder-decoder temporal convolutional network architecture that the proposed ED-TCN adapts from action segmentation.","marker":"[32]"},{"why":"Introduces dilated convolutions, which the model uses to enlarge temporal context in the encoder and decoder.","marker":"[43]"},{"why":"Introduces circular convolution padding, the basis of the periodic padding used on chromagram pitch-class rows.","marker":"[42]"},{"why":"Provides the separate real-world concert recordings used to test generalization after fine-tuning.","marker":"[15]"},{"why":"Supplies the vocal-stem separation used to extract singing from polyphonic concert audio before evaluation.","marker":"[44]"},{"why":"Establishes the collar-based evaluation convention (200 ms) used to score event boundaries.","marker":"[34]"}],"fun_headline_variants":["Don't-care labels lift Indian vocal ornament F1 by 7-8 points","ED-TCN with periodic padding outdoes CRNN on raga ornaments","Active annotation yields ROD dataset for six Hindustani ornaments","Boundary-preserving chunking improves ornament detection in vocal music","Human-in-the-loop labels power deep ornament detection in Indian classical"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported results assume the test audio is chunked with knowledge of the true ornament boundaries, so a truncated ornament never appears at test time; the paper does not describe how the model would be applied to long recordings when those boundaries are unknown.","fun_headline_variants_meta":{"raw":{"variants":["Don't-care labels lift Indian vocal ornament F1 by 7-8 points","ED-TCN with periodic padding outdoes CRNN on raga ornaments","Active annotation yields ROD dataset for six Hindustani ornaments","Boundary-preserving chunking improves ornament detection in vocal music","Human-in-the-loop labels power deep ornament detection in Indian classical"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1405,"prompt_tokens":984,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":600,"tokens_out":421,"duration_ms":4235,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:29:52.314156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained model and score it on the ROD test files cut into fixed 10-second windows with no access to event boundaries, then compare $F_1$ to the event-aware-chunking result; if performance falls toward the baseline, the reported gains are conditional on knowing where ornaments start.","supporting_citations":[{"cited_title":"Analysis and detection of singing techniques in repertoires of j-pop solo singers,","cited_arxiv_id":null,"evidence_quote":"Supplies the CRNN baseline and the frame-wise singing-technique detection formulation this work compares against."},{"cited_title":"Circular convo- lutional neural networks for panoramic images and laser data,","cited_arxiv_id":null,"evidence_quote":"Introduces circular convolution padding, the basis of the periodic padding used on chromagram pitch-class rows."},{"cited_title":"Explainable Deep Learning Analysis for Raga Identification in Indian Art Music","cited_arxiv_id":"2406.02443","evidence_quote":"Provides the separate real-world concert recordings used to test generalization after fine-tuning."}],"review_version":1}