{"id":"ee1df1fc-26f3-4b85-8820-d552a56cf3d8","arxiv_id":"2411.18611","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A system using Monte-Carlo dropout for OOD detection and contrastive novel class discovery clusters unseen raga classes from unlabeled audio with 79-81% clustering accuracy on benchmark datasets.","lead":"This paper pairs uncertainty-based out-of-distribution detection with contrastive clustering to identify and group raga classes never seen during training. If it works, music-analysis systems could grow their known raga vocabulary from unlabeled recordings instead of requiring expert-labeled data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clustering accuracy may be inflated by recording/artist identity rather than raga semantics; the paper's own Saraga Shree/Marwa failure is evidence of this confound.","rationale":"The reader identifies transferability of ffeat as the weakest assumption; I agree but sharpen the risk. The danger is not simply that the features fail to transfer, but that what transfers is singer or recording-session identity, which is correlated with raga labels in these small datasets. The paper's own observation that Shree and Marwa cluster together due to shared singers and concert is direct evidence that the embedding space can be dominated by non-ragacovariates. This is load-bearing because the headline claim is that clusters correspond to unseen raga classes. The feature ablation (Table 3) does not resolve it because ffeat is tested on its own training set Sl, while the Su results are contaminated by the confound. The proposed artist-split test would settle the question by measuring whether clustering generalizes across performers. The loss-component ablation and openness study are useful, and the paper is internally coherent, so a conditional verdict is appropriate: accept if the artist-split evaluation confirms raga-driven clustering, and otherwise the central claim is not established.","tokens_in":10373,"tokens_out":7899,"duration_ms":173948,"concrete_test":"Use the artist/concert metadata in PIM and Saraga to split the OOD test clips into two non-overlapping artist sets, ensuring each raga appears in both sets. Train ffeat on the 12 labeled PIM classes as in the paper, then run the complete proposed pipeline (same hyperparameters, K=5, and threshold selection) separately on (i) clips from artist set A and (ii) clips from artist set B. Compute k-means ACC with standard Hungarian matching for each subset and compare to the reported full-set ACC. If ACC in the held-out artist subsets drops substantially (e.g., more than 10 points) or approaches chance, the clusters are confounded by recording or artist identity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the NCD clusters correspond to previously unseen raga classes. The load-bearing premise is that the ffeat embeddings in Eq. (2)-(3) separate unseen ragas by melodic content. This premise is unproven because ffeat was trained on only 12 PIM classes, and the only validation for unseen classes is the end-to-end clustering result itself. The paper's feature ablation (Table 3) is not probative: ffeat is evaluated on the same labeled set Sl on which it was supervised, reaching ACC 90, whereas MERT and melody features were never trained on Sl. More importantly, the embedding space may encode covariates that are correlated with raga labels in these small datasets, such as singer identity, concert, or recording conditions. The paper itself provides evidence: in Section 4.3.1, Raag Shree and Raag Marwa are described as sharing the same singers and the same concert; the method clusters them together despite their being distinct ragas. That is exactly the confound that would inflate clustering accuracy if each raga class in the test set is represented by a small number of artists or recordings. Because K is set to the known number of unseen classes and the evaluation ACC uses a one-to-one matching rule that invalidates results when one predicted cluster is shared, the reported 79-81% ACC may overstate the semantic quality of the clusters. If the embeddings are primarily separating recording sessions, the pipeline is not discovering ragas; it is discovering recording identities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for open-set raga identification: MC-dropout uncertainty is used to detect out-of-distribution audio clips, and a contrastive-learning-based Novel Class Discovery (NCD) module clusters the OOD clips into unseen raga classes. A CNN-LSTM feature extractor f(·) is supervised on 12 known PIM ragas; its penultimate embeddings ffeat(·) are then used to generate pairwise cosine-similarity pseudo-labels (Eqs. 2–3) that train a self-attention encoder g(·) under a combined loss of BCE, contrastive, and consistency terms (Eq. 10). The reported results include OOD accuracy of 85.6% (Saraga) and 80.87% (PIM), and K-means clustering ACC of 79.34% (PIM) and 81.04% (Saraga) for the proposed method versus 70.75% and 75.44% for the baseline (Table 4). The paper also provides loss-component ablations, an openness study, and releases code and metadata.","tokens_in":10753,"tokens_out":3880,"duration_ms":39755,"significance":"If the main claim holds, the paper would make a useful contribution to MIR: it formulates unseen-raga discovery as NCD, defines an openness measure adapted to the problem, and demonstrates a fully unsupervised clustering pipeline that improves over direct clustering of the pretrained embeddings. The release of code and the inclusion of cross-dataset experiments on PIM and Saraga are concrete strengths. However, the significance is tempered by validation gaps: the feature ablation that motivates ffeat(·) compares it against features that were not trained on the same labeled set, the pseudo-label generation is partly self-referential, no error bars or statistical tests are reported, and the paper itself provides evidence that recording/artist identity may drive the Saraga Shree/Marwa confusion. These issues affect the interpretation of the central clustering claim, so the contribution is promising but not yet fully established.","major_comments":[{"comment":"The load-bearing premise is that ffeat(·), supervised only on the 12 PIM training ragas, produces embeddings in which entirely unseen raga classes are already separable by cosine similarity. The feature ablation in Table 3 does not establish this premise: ffeat(·) is evaluated on the same labeled set Sl on which it was supervised, reaching ACC 90.05, whereas MERT and melody features were never trained on Sl. A fair transfer test would cluster ffeat(·) embeddings on Su (the unseen raga set) and compare them with frozen general-purpose features on the same Su. As written, the choice of ffeat(·) is supported by a biased comparison, and the pseudo-labels in Eq. (3) inherit whatever structure ffeat(·) has on unseen classes.","section":"Section 3.3.1 and Section 4.2, Eqs. (2)–(3)"},{"comment":"There is no end-to-end evaluation of the complete OOD+NCD pipeline. OOD detection is reported separately in Table 2, and NCD in Table 4 is evaluated on a Su that is assumed to consist entirely of OOD samples. In a real deployment, OOD detection errors would propagate into the clustering stage. I ask for a combined evaluation that first runs MC-dropout OOD detection on a mix of known and unknown classes and then clusters only the samples flagged as OOD, reporting clustering accuracy over the true unseen-class subset and the overall pipeline accuracy.","section":"Section 3.3 and Section 4.3"},{"comment":"The paper states that Raag Shree and Raag Marwa are clustered together partly because the recordings feature the same singers and come from the same concert. This is direct evidence that the embeddings may be capturing recording/artist identity rather than raga semantics. Because each Saraga unseen class is represented by at most 3 audio files, a small number of artists or sessions could drive the reported 81.04% ACC. The authors should control for this confound by, for example, splitting by artist/session, reporting per-artist clustering performance, or removing same-concert recordings and re-running Table 4.","section":"Section 4.3.1, Saraga discussion"},{"comment":"The ACC evaluation rule invalidates the result if one predicted cluster is matched to multiple true clusters, and the reported numbers give no indication of how often this occurred or how many clusters were discarded. A one-to-one matching rule can make ACC optimistically high relative to a full contingency-table matching. The authors should report the number of valid runs, use a standard Hungarian matching over the full confusion matrix, or at least report ARI as a primary label-dependent metric alongside ACC.","section":"Section 3.5 and Table 4"},{"comment":"The OOD variance threshold and the similarity threshold δ are effectively free parameters, yet no sensitivity analysis is provided and the text does not say how the thresholds were chosen. If they are tuned on the very test sets used in Tables 2 and 4, the reported numbers are optimistic. Please report threshold values, show performance across a range of thresholds, or describe a validation-based selection procedure that does not use the final test classes.","section":"Section 3.2, Section 3.3.1, and Section 4.1"},{"comment":"The claim that the proposed clusters 'can approach or match the clustering quality of supervised methods' is not supported by any experiment: Table 4 compares only against unsupervised baselines on Su, and no supervised upper bound for the same unseen classes is reported. Adding a supervised clustering result (for example, k-means on embeddings from a model fine-tuned with labels on Su, or a linear-probe accuracy) would make this claim testable. Until then, the statement should be removed or substantially weakened.","section":"Section 4.3.1, final paragraph"}],"minor_comments":[{"comment":"The notation is inconsistent: the text defines hard negatives as ξh, but Eq. (9) uses ξm, and Algorithm 1 line 16 uses β for the set of positive embeddings while β is also a loss weight in Eq. (10). Please unify these symbols.","section":"Section 3.3.3, Eq. (9)"},{"comment":"The notation Cl and Cu appears to denote both class counts and class sets; the summation limits should be |Cl| and |Cu| or the sets themselves, and the sums appear to run over samples rather than classes, so the expression as written is unclear.","section":"Section 3.3.2, Eq. (6)"},{"comment":"The definition of cpt is ambiguous: it should state clearly that the intersection is over the sets of embeddings assigned to the predicted cluster and the true cluster, and explain how the 'highest overlap' match is computed.","section":"Section 3.5, ACC definition"},{"comment":"The OOD detection section does not report the variance threshold value, the number of threshold trials, or a confusion matrix for the OOD decision; adding these would improve reproducibility.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a music information retrieval venue and the central idea is worth pursuing, but the current evidence does not yet rule out the artist/recording confound that the authors themselves describe for Shree/Marwa. I would encourage the editor to request the controls and end-to-end experiments listed in the major comments rather than rejecting the paper outright. The code-release promise is a positive signal, but verifying that the released repository contains the full training and evaluation scripts would also be useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is applying novel class discovery to unseen raga classes, with MC-dropout OOD detection feeding a contrastive encoder that improves over direct clustering baselines. The reported numbers are believable as numbers, but the paper hasn't yet shown the clusters correspond to raga semantics rather than recording or singer identity.\n\nThe paper does several things well. The domain-specific adjustments—same-source positive pairs, time-shift and volume transforms for the consistency loss, chroma features—are sensible and well-motivated. The openness study is a useful addition. The authors also openly report the Shree/Marwa failure, which is more than most papers do.\n\nThe main weakness is in the evaluation. The feature ablation compares ffeat, which was supervised on Sl, against MERT and melody features that were never trained on Sl; that is not an apples-to-apples test. More importantly, the clustering result could be inflated by non-musical covariates. The paper itself notes that Shree and Marwa are clustered together and that those recordings share the same singers and the same concert. With at most three audio files per class in Saraga, the encoder could be separating recording sessions rather than melodic structure. The ACC metric also assumes a one-to-one cluster-to-class mapping, which is forgiving when K is set to the known number of classes. There are no error bars anywhere, so it is unclear whether the margins in Table 4 are stable.\n\nThis is a serious, readable paper for the MIR community. It deserves peer review, but a revision should include: an unbiased feature ablation where all extractors are evaluated on the same held-out set, an end-to-end evaluation that chains OOD detection into NCD, error bars across runs, and a clustering evaluation that controls for singer or recording identity—for example, reporting within-recording versus across-recording cluster agreement. Without those, the semantic discovery claim remains a plausible hypothesis, not an established result.\n\nI would send it to peer review and tell the authors that the main barrier to acceptance is ruling out the recording-identity confound, not the method itself.","headline":"A serious NCD-for-ragas paper with a real new application, but the clustering evaluation doesn't fully rule out recording-identity confounds, so the semantic claim needs a stronger test.","tokens_in":11231,"tokens_out":2004,"would_cite":true,"duration_ms":21593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a two-stage pipeline that uses MC-dropout uncertainty to detect out-of-distribution raga audio and contrastive Novel Class Discovery to cluster those unseen ragas, reporting clustering accuracy of 79.34% on PIM and…","keywords":["raga identification","open-set recognition","out-of-distribution detection","novel class discovery","contrastive learning","Monte Carlo dropout","Indian art music","music information retrieval"],"falsifier":"An experiment that would settle it: train the same pipeline but replace $f_{feat}$ with a feature extractor trained on scrambled or non-rag audio, then cluster the same five unseen ragas; if clustering accuracy stays near chance, the transfer premise is doing the work. More directly, compute the cosine similarity of $f_{feat}$ embeddings for pairs of clips from the same unseen raga and from different unseen ragas; the method requires that a threshold $\\delta$ separates these two distributions, so measuring the overlap of those two distributions would show whether the premise holds.","tokens_in":10232,"feed_emoji":"🎵","tokens_out":6878,"duration_ms":57416,"temperature":0.7,"pith_summary":"This paper argues that raga identification should be treated as an open-set problem, because new ragas can appear at test time. It proposes a two-stage pipeline: first, Monte Carlo dropout uncertainty flags out-of-distribution audio clips; second, a Novel Class Discovery stage clusters those clips into distinct unseen raga classes using contrastive learning on unlabeled data. With a CNN-LSTM feature extractor trained on 12 labeled PIM ragas, k-means on the discovered embeddings reaches 79.34% clustering accuracy on PIM and 81.04% on Saraga, close to the clustering quality of supervised methods. The significance is that this would allow music identification systems to use vast unlabeled recordings without expert annotation.","feed_headline":"Unseen ragas cluster to 81% accuracy without labels","feed_subtitle":"MC-dropout flags unknown ragas, then contrastive learning groups them — no expert annotation needed.","key_machinery":"The carrying mechanism is the combined training objective for the encoder $g(\\cdot)$, with pseudo-labeled BCE loss $\\ell_{bce}$, hard-negative contrastive loss $\\ell_{cl}$, and consistency MSE loss $\\ell_{mse}$. Pairwise pseudo-labels $t_{i,j}$ are obtained by thresholding the cosine similarity $\\varepsilon$ of $f_{feat}$ embeddings (Eq. 3), so the labeled structure of known ragas is transferred to unlabeled data. Hard negatives are the $H$ least similar embeddings, and the consistency loss ties a clip to time-shifted and volume-modified versions of itself. The openness of the NCD problem is quantified by $O_{NCD} = 1 - \\sqrt{2|C_{train}| / (2|C_{train}| + |C_{test}|)}$, and the ablation shows each loss component contributes to the final clustering quality.","core_discovery":"The paper's central claim is that a representation trained only on known ragas carries enough melodic structure that entirely unseen raga classes can be recovered by clustering. The authors train the feature extractor $f_{feat}$ on 12 PIM ragas, then train a self-attention encoder $g$ on unlabeled OOD samples with the combined loss $\\ell = \\ell_{bce} + \\beta\\ell_{cl} + \\gamma\\ell_{mse}$, where cosine-similarity-based pseudo-labels provide BCE supervision, hard negatives provide contrastive supervision, and time/volume shifts provide consistency supervision. The resulting embeddings cluster the five unseen ragas in each dataset with an ACC of 79.34% (PIM) and 81.04% (Saraga) using k-means, outperforming baselines that cluster the raw features directly. The paper also reports that the approach generalizes across datasets, with features trained on PIM clustering Saraga audio successfully.","pith_inferences":["Beyond the paper, the pseudo-label threshold $\\delta$ is a free parameter that could be tuned per-dataset or per-desired cluster granularity, since no principled method for setting it is proposed.","Beyond the paper, the pipeline should be most reliable when unseen ragas are melodically close to the training ragas; a targeted test that varies the thaat distance between seen and unseen ragas would map where transfer breaks.","Beyond the paper, the same OOD-plus-contrastive scheme could be applied to other open-set music discovery tasks such as tala or instrument family discovery, since the method is feature-agnostic.","Beyond the paper, the similarity-based pseudo-labels may be the main source of confusion for Mishra ragas, which mix notes from multiple parent ragas and produce the lowest per-class F1 scores in the confusion matrix."],"forward_implications":["If the central claim is correct, unseen ragas can be grouped from unlabeled audio at roughly 79–81% clustering accuracy, reducing the need for expert-labeled data.","OOD detection accuracy of 80.87% (PIM) and 85.6% (Saraga) implies that known and unknown ragas can be separated without retraining the classifier.","The combined loss beats each component alone (ACC 79.34% vs 76.04% for $\\ell_{cl+bce}$, 75.43% for $\\ell_{bce}$, 70.16% for $\\ell_{cl}$), so the interaction of the three terms is doing real work.","Higher openness degrades performance: moving from $O_{NCD}=0.09$ to $0.18$ drops ACC from 79.34% to 55.68%, so the method's usefulness is bounded by how many unseen classes coexist."],"supporting_citations":[{"why":"Supplies MC-dropout as the Bayesian approximation used to compute predictive variance for OOD detection.","marker":"[14]"},{"why":"Provides the Neighborhood Contrastive Learning framework whose contrastive loss is adapted to raga embeddings.","marker":"[17]"},{"why":"Defines the PIM labeled dataset and the supervised 12-raga classification task used to train the feature extractor.","marker":"[6]"},{"why":"Supplies the Saraga Hindustani dataset used as a cross-dataset unseen target for OOD detection and clustering.","marker":"[8]"},{"why":"Introduces the open-set recognition formulation whose openness measure is adapted to define NCD openness.","marker":"[7]"},{"why":"Explains why contrastive loss alone can distort semantic structure, motivating the combination with BCE and consistency losses.","marker":"[33]"}],"fun_headline_variants":["Clustering unseen ragas hits 81% accuracy with no labels","Open-set raga ID: contrastive learning clusters novel classes","Label-free clustering of unseen ragas reaches 81% ACC","Detect out-of-distribution ragas, then cluster them to 81%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the CNN-LSTM feature extractor trained only on the 12 known PIM ragas produces embeddings in which entirely unseen raga classes are already separated by cosine similarity; if that transfer fails, both the pseudo-labels and the final clusters degrade.","fun_headline_variants_meta":{"raw":{"variants":["Clustering unseen ragas hits 81% accuracy with no labels","Open-set raga ID: contrastive learning clusters novel classes","Label-free clustering of unseen ragas reaches 81% ACC","Detect out-of-distribution ragas, then cluster them to 81%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3088,"prompt_tokens":871,"completion_tokens":2217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2150}},"tokens_in":487,"tokens_out":2217,"duration_ms":15464,"temperature":1.0,"reasoning_tokens":2150,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:00:10.781175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that would settle it: train the same pipeline but replace $f_{feat}$ with a feature extractor trained on scrambled or non-rag audio, then cluster the same five unseen ragas; if clustering accuracy stays near chance, the transfer premise is doing the work. More directly, compute the cosine similarity of $f_{feat}$ embeddings for pairs of clips from the same unseen raga and from different unseen ragas; the method requires that a threshold $\\delta$ separates these two distributions, so measuring the overlap of those two distributions would show whether the premise holds.","supporting_citations":[{"cited_title":"Toward open set recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies MC-dropout as the Bayesian approximation used to compute predictive variance for OOD detection."},{"cited_title":"Simple and scalable predictive uncertainty estimation using deep ensembles,","cited_arxiv_id":null,"evidence_quote":"Provides the Neighborhood Contrastive Learning framework whose contrastive loss is adapted to raga embeddings."},{"cited_title":"Explainable deep learning analysis for raga identification in indian art music,","cited_arxiv_id":null,"evidence_quote":"Defines the PIM labeled dataset and the supervised 12-raga classification task used to train the feature extractor."},{"cited_title":"Phononet: multi-stage deep neural net- works for raga identification in hindustani classical music,","cited_arxiv_id":null,"evidence_quote":"Supplies the Saraga Hindustani dataset used as a cross-dataset unseen target for OOD detection and clustering."},{"cited_title":"The computational study of a musical cul- ture through its digital traces,","cited_arxiv_id":null,"evidence_quote":"Introduces the open-set recognition formulation whose openness measure is adapted to define NCD openness."},{"cited_title":"S3t: Self-supervised pre-training with swin transformer for music classification,","cited_arxiv_id":null,"evidence_quote":"Explains why contrastive loss alone can distort semantic structure, motivating the combination with BCE and consistency losses."}],"review_version":1}