{"id":"8b5dc38e-f813-41bc-9d9c-b688ff3fcce6","arxiv_id":"2411.14907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A platform for comparing AI music embeddings with human similarity judgments, applied to Carnatic percussion duets, finds only 66.66% agreement between NSynth and an expert's rhythmic-harmony ratings.","lead":"DAIRHuM is a new software platform that lets musicians label how similar music tracks are, then checks whether an AI model makes the same judgment. The authors test it on Carnatic percussion duets and find the NSynth model agrees with an expert on rhythmic harmony only about 66% of the time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Alignment score 66.66% is inflated: counting all 36 cells of Table III includes trivial self-pairs; with only the 15 off-diagonal pairwise comparisons it is 9/15 = 60%, so the headline quantitative claim is unsupported as reported.","rationale":"The reader's conditional verdict is appropriate, and the reader correctly flagged the unstated conversion of human ratings into pairwise comparisons. However, the more concrete and load-bearing problem is that the reported alignment score itself is internally inconsistent with the table shown: 66.66% corresponds to counting all 36 cells, including six trivial diagonal self-comparisons, whereas the 15 meaningful pairwise comparisons give only 60% (9 matches out of 15). This numerical discrepancy directly undermines the abstract's claim of 'significant findings on model alignment with human judgments of rhythmic harmony' and the specific 'alignment score of 66.66%' highlighted in the paper. My reconstruction shows that the off-diagonal entries of Table III can be obtained by converting the per-track labels of Table I via exact equality, but this rule is nowhere stated, making the human ground truth labels non-reproducible. No baselines, error bars, or multiple-expert agreement measures are provided, so even a corrected score would need additional support to justify 'significant findings.' The platform itself is a plausible contribution, and the code is public, so a revised evaluation could address these issues; hence the verdict should remain conditional rather than reject. I do not find evidence of intentional misreporting; the diagonal inclusion and unstated conversion appear to be methodological oversights, but they are consequential for the central quantitative claim.","tokens_in":5583,"tokens_out":6582,"duration_ms":58050,"concrete_test":"Recompute the alignment score from Table III excluding all diagonal cells, or better from the off-diagonal upper triangle only; if the score drops from 66.66% to 60% as the table implies, revise the abstract and Section III.C accordingly. Also state the rule used to convert Table I ratings to pairwise same/different labels and check whether the 60% score changes under plausible alternative rules (e.g., treating adjacent labels as 'same' vs requiring exact label equality).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table III is a 6x6 matrix of M/NM for pairwise sameness. The diagonal is trivially M because every track is identical to itself. The reported 66.66% equals 24/36, i.e., all cells including the diagonal. Restricting to the 15 distinct non-diagonal pairs (or 30 directed off-diagonal entries) yields 9/15 = 18/30 = 60%. The 66.66% figure therefore depends on counting self-comparisons, which cannot provide evidence about alignment. In addition, the human side of Table III is not directly observed: it is reconstructed from the per-track labels in Table I by treating equal labels (e.g., S vs S) as 'same' and different labels as 'different' (Sections III.A and III.C). That conversion rule is never stated, and it is the only way to obtain the non-match pattern in the table. Because the abstract's 'significant findings' rest on this score, the quantitative central claim is not currently supported. The broad platform contribution may still be valid, but the paper needs a corrected score without diagonal entries, an explicit statement of the label-to-pairwise conversion, and ideally a sensitivity check under alternative conversions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"DAIRHuM is an open-source Python platform designed to compare AI-model representations of music with pairwise human similarity judgments. Its workflow is: (i) users label source recordings and their variations with perceptual labels; (ii) a pre-trained model (here NSynth) generates embeddings for each track; (iii) a permutation test with a distance metric (here MMD with RBF kernel) converts embedding distances into binary 'distinguishable/indistinguishable' judgments; and (iv) these are compared with human labels to produce an alignment score and plots. The paper applies this pipeline to three Carnatic percussion duets (Mridangam and Kanjira) with labels from the author's earlier study [9] and reports a 66.66% alignment score in Table III, together with a qualitative discussion of human/model differences in the organization of rhythmic space. The main quantitative claim is currently not supported because the score counts trivial self-pairs and the human pairwise labels are reconstructed from per-track ratings by an unstated rule.","tokens_in":5868,"tokens_out":12001,"duration_ms":113565,"significance":"The platform addresses a genuine gap: it offers a simple, embedding-agnostic way to benchmark music AI models against human judgments in genres outside the Western mainstream, and the public code makes experimentation accessible. The emphasis on a culturally specific, low-resource genre is timely, and the idea of using pre-trained embeddings to avoid genre-specific feature engineering is practical. However, the empirical demonstration is a case study rather than a benchmark: it uses one expert rater, one model, one small dataset, and a single score without uncertainty or a chance baseline. The reported alignment score cannot currently be interpreted, so the paper's abstract-level claim of 'significant findings' is not established. The conceptual and software contribution remains defensible if the quantitative analysis is corrected and reframed.","major_comments":[{"comment":"The alignment score of 66.66% is computed over all 36 cells of Table III, including the six diagonal self-pairs. A track compared with itself is trivially 'same' for both model and human, so the diagonal contributes six forced matches. Excluding the diagonal leaves 18 matches out of 30 directed off-diagonal entries (9 of 15 undirected pairs), i.e., 60%, not 66.66%. Because the Abstract and Section IV rely on this score, please recompute it, report diagonal-excluded results separately, and state which recording (R1, R2, or R3) Table III refers to or how scores are aggregated across songs.","section":"Section III.C, Table III"},{"comment":"The human side of Table III is not directly observed. Table I provides per-track ordinal labels ('mostly', 'half the time', 'sometimes', 'always'), while Table III reports pairwise same/different agreement, so a conversion from labels to pairs is needed but never stated. If the intended rule is 'equal labels are same and different labels are different', this binarization is one of several plausible mappings and could change the computed alignment score. Please state the conversion explicitly and report the sensitivity of the score to alternative binarizations.","section":"Section III.A and III.C, Tables I and III"},{"comment":"The reported score has no uncertainty and no chance baseline. With 30 directed off-diagonal comparisons, 18/30 = 60% is not significantly above chance under a two-sided binomial test (p ≈ 0.36); even the reported 24/36 including diagonal is close to the 21/36 expected if off-diagonal matches were random. Please report confidence intervals, compare against a chance or majority-class baseline, and give per-song scores before claiming 'significant findings' in the Abstract.","section":"Section III.C"},{"comment":"All human ratings come from a single expert and from the author's own earlier study [9]; no inter-rater reliability or independent validation is reported. As a result, the application cannot support the general statement that the model aligns with 'human judgments of rhythmic harmony'; it is at best a case study with one rater. Please report the number of raters in [9], any available reliability statistics, and temper the Abstract and Section IV claims accordingly.","section":"Section III.A"},{"comment":"The platform description encourages users to 'experiment with a selection of distance metrics' and 'find the one that aligns most closely with their chosen labeling scheme', and the adjustable settings include kernel function, gamma, number of permutations, and p-value threshold. If these settings were chosen after inspecting the same human labels that are later scored in Table III, the reported alignment is not a confirmatory result. Please state whether the NSynth-MMD-RBF configuration was fixed before computing Table II and Table III, and report how the alignment score varies across a small grid of reasonable settings.","section":"Section III.B"}],"minor_comments":[{"comment":"The caption 'System/Human' is ambiguous; clarify that rows and columns are track indices and that each entry records whether the model and human agree on sameness for that pair.","section":"Table III caption"},{"comment":"Figure 1 is referenced in Section III.C but no figure content appears in the manuscript; please include it and add readable axis labels and a legend.","section":"Figures"},{"comment":"The symbol M is used for both 'mostly' (Table I) and 'match' (Table III); use distinct labels to prevent confusion.","section":"Tables I and III"},{"comment":"To make the experiment reproducible, specify the NSynth model version/checkpoint and embedding extraction details, including how each track is chunked or pooled, and the exact formula for the RBF gamma (e.g., reciprocal of median pairwise distance).","section":"Section III.B"},{"comment":"Please provide or cite the generative procedure from [9] that created the five Kanjira variations, and indicate whether the audio files and ratings are released alongside the platform code.","section":"Section III.A"},{"comment":"The phrase 'significant findings' should be qualified (e.g., 'a case study showing partial alignment') unless the statistical concerns in the major comments are resolved.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core idea — a lightweight embedding-based alignment platform for culturally specific music — is worth publishing in revised form. My main concern is that the quantitative demonstration is less solid than the abstract suggests, and the paper should be judged primarily as a software/tool paper backed by an illustrative case study rather than as a benchmark result. The first-platform claim is plausible but not deeply established; a comparison with general-purpose similarity-judgment tools would help. I recommend major revision rather than rejection because the issues are fixable: recompute the score excluding self-pairs, state the label-to-pairwise conversion, and add uncertainty and chance baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The platform idea is worth keeping, but the headline quantitative claim does not survive contact with Table III. The 66.66% alignment score includes the six diagonal cells, where a track is compared to itself; those are trivial matches. Restricting to the 15 distinct off-diagonal pairs gives 9/15, or 60%. That is a material drop, and the abstract's 'significant findings' rest on the inflated number.\n\nWhat is genuinely new: DAIRHuM as a packaged, open-source tool. The workflow is sensible—label tracks, embed them, compare pairwise distances to human labels—and the Carnatic percussion duet case is a good illustration for an under-represented genre. Using MMD/Wasserstein with permutation tests is standard but appropriate. The platform could be genuinely useful for researchers who want a quick way to probe cultural sensitivity in embeddings.\n\nThe soft spots are concentrated in the evaluation. First, the diagonal issue above. Second, the conversion from per-track labels like 'mostly' and 'sometimes' to a binary same/different judgment is never specified. That is a load-bearing omission: without the rule, Table III is not reproducible. Third, the empirical evidence is thin—one expert, one model, one small dataset, no chance baseline, no error bars. A random baseline would give around 50% alignment, so 60% is not obviously meaningful without a confidence interval. Fourth, the human labels come from the author's earlier study, which makes the evaluation self-referential; that is not disqualifying, but it does reduce the sense of independent validation.\n\nThe paper is not incoherent, and the platform itself deserves engagement. As submitted, though, the empirical claims are under-supported. A serious referee should ask for a corrected score without diagonal self-pairs, an explicit statement of the label-to-pairwise conversion, a chance baseline, and ideally a second expert. The code is public, so these fixes are straightforward. I would send it to peer review, but with the expectation of major revision.","headline":"Useful platform, but the 66.66% alignment score is inflated by counting trivial diagonal self-matches, and the paper never states how per-track human labels become pairwise comparisons.","tokens_in":6349,"tokens_out":3775,"would_cite":false,"duration_ms":34880,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DAIRHuM is a platform for scoring whether a pretrained music model's embeddings agree with expert musical judgments, and applying it to Carnatic percussion duets gives a 66.66% alignment score.","keywords":["human-AI alignment","music information retrieval","Carnatic music","rhythmic harmony","NSynth embeddings","source-variation labeling","permutation test","Mridangam-Kanjira duet"],"falsifier":"Re-run the platform's scoring on the same 18 tracks with a second expert's ratings, using an explicit rule to map those ratings to pairwise same/different labels; if the resulting alignment score differs substantially from 66.66%, the reported result depends on the particular expert's labeling rather than a stable property of the model.","tokens_in":5410,"feed_emoji":"🎵","tokens_out":7167,"duration_ms":65689,"temperature":0.7,"pith_summary":"DAIRHuM is a Python platform that lets musicians and experimenters organise audio tracks into sources and musical variations, generate embeddings from a pretrained model, and get a percentage score for how often the model's pairwise same/different judgments agree with human labels. Applying it to a Carnatic percussion dataset consisting of three original Mridangam-Kanjira duets plus five synthetic Kanjira variations each, the paper reports an alignment score of 66.66% between NSynth embeddings and expert ratings of rhythmic harmony. The point of the platform is to make human-AI alignment assessment practical for genres with scarce annotations, using only audio, embeddings, and human labels rather than genre-specific annotation pipelines. If the platform works as claimed, music researchers can benchmark any pretrained model against expert perception in under-represented musical traditions without building bespoke datasets.","feed_headline":"AI embeddings match a Carnatic expert's harmony ratings 66.66 percent","feed_subtitle":"DAIRHuM lets musicians label track similarity and score a pretrained model's agreement against an expert ear.","key_machinery":"The machinery is a three-stage alignment pipeline. First, a source-variation labelling scheme organises recordings into originals and variations labeled by a human-judged musical attribute, here the degree of rhythmic harmony between the Mridangam and Kanjira. Second, a pretrained model (NSynth) produces embeddings, and pairwise permutation tests with a choice of metrics (MMD with an RBF kernel and gamma set to 1/median distance, or Wasserstein distance) convert embedding distances into binary same/different judgments using p<0.05 as the threshold. Third, an alignment score counts matches between the model's binary labels and the expert's labels across all pairwise source-variation comparisons, yielding the reported 66.66%. The platform is intentionally agnostic to genre-specific representations: any audio collection, any embeddings, and any human labels can feed the same scoring procedure.","core_discovery":"The paper's central claim is that DAIRHuM is the first platform designed to directly measure alignment between AI music representations and human musical judgments in a culturally specific genre, and that applying it to Carnatic percussion duets reveals significant findings on model alignment with human judgments of rhythmic harmony. In the reported study, a Carnatic expert rated each source recording and each synthetic Kanjira variation as 'mostly', 'half the time', or 'sometimes' rhythmically congruent with the Mridangam, and the NSynth model's embeddings were then compared pairwise using the MMD metric with a 1000-iteration permutation test, labeling pairs with p<0.05 as distinguishable. The paper reports an alignment score of 66.66% between human and model labels, and the qualitative plots show that while both group some variations with the same harmony rating together, the model generally separates variations into finer, more distinct categories than the expert does.","pith_inferences":["If the conversion from ordinal labels like 'mostly', 'half the time', and 'sometimes' to pairwise same/different judgments were written down and made public, the platform could pool ratings from multiple experts, turning the 66.66% score into a distribution with a confidence interval rather than a point estimate.","The DAIRHuM scoring procedure could serve as a model-selection diagnostic: a researcher with labeled data could compare candidate embedding models and pick the one whose pairwise judgments best match expert labels.","The paper's qualitative observation that humans and models both group some variations together suggests a testable extension: measure clustering in embedding space against clusterings inferred from expert ratings, rather than only binary pairwise agreement.","Because the expert ratings are per-track congruence labels and not explicit pairwise judgments, a follow-up experiment collecting direct pairwise same/different ratings from several Carnatic musicians would separate the platform's scoring method from the particular rating scale used in this dataset."],"forward_implications":["If DAIRHuM's alignment score is a valid measure, a pretrained model like NSynth can be benchmarked against expert rhythmic-harmony perception in Carnatic music without any genre-specific training data.","The 66.66% alignment figure implies that NSynth captures a substantial but incomplete share of one expert's same/different judgments, motivating culturally aware training data or post-hoc calibration.","Because the platform only needs audio, embeddings, and human labels, the same procedure transfers to other low-resource genres, making alignment testing accessible to musicians without machine-learning expertise.","The model's tendency to split variations into more fine-grained categories than the expert suggests a concrete failure mode: embedding distinguishability need not correspond to perceived harmonic similarity.","The generated plots give qualitative checks of where human and model judgments diverge, such as transitions where changes in accompaniment shift an expert's category but leave the model's rating unchanged."],"supporting_citations":[{"why":"Supplies the Kanjira source-variation dataset and the expert harmony ratings against which the reported alignment score is computed.","marker":"[9]"},{"why":"The benchmark platform whose design DAIRHuM adapts, and the comparison that supports the 'first platform for culturally specific music alignment' claim.","marker":"[10]"},{"why":"Demonstrates alignment of generative-model surprise with human enjoyment, the alignment problem DAIRHuM imports into music information retrieval.","marker":"[1]"},{"why":"Shows predictive models aligning with human auditory segmentation, providing a prior example of model-human alignment that this work extends.","marker":"[2]"},{"why":"Establishes computational rhythm analysis of Indian classical music and frames the genre-specific challenge DAIRHuM targets.","marker":"[4]"},{"why":"Documents the difficulty of applying music deep learning to traditional music, the cultural-specificity gap DAIRHuM addresses.","marker":"[3]"}],"fun_headline_variants":["Carnatic drum AI matches expert ear 66.66% of the time","First platform scores AI harmony alignment in Carnatic music","AI vs expert: 66.66% agreement on Carnatic rhythmic harmony","DAIRHuM: New tool measures AI-human musical harmony alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire alignment score rests on how the single, unnamed expert's per-track labels ('mostly', 'half the time', 'sometimes') were converted into the pairwise same/different judgments used in Table III, and that conversion rule is not specified in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Carnatic drum AI matches expert ear 66.66% of the time","First platform scores AI harmony alignment in Carnatic music","AI vs expert: 66.66% agreement on Carnatic rhythmic harmony","DAIRHuM: New tool measures AI-human musical harmony alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2429,"prompt_tokens":934,"completion_tokens":1495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1418}},"tokens_in":550,"tokens_out":1495,"duration_ms":11003,"temperature":1.0,"reasoning_tokens":1418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:43:30.793818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the platform's scoring on the same 18 tracks with a second expert's ratings, using an explicit rule to map those ratings to pairwise same/different labels; if the resulting alignment score differs substantially from 66.66%, the reported result depends on the particular expert's labeling rather than a stable property of the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kanjira source-variation dataset and the expert harmony ratings against which the reported alignment score is computed."},{"cited_title":"Schrimpf, J","cited_arxiv_id":null,"evidence_quote":"The benchmark platform whose design DAIRHuM adapts, and the comparison that supports the 'first platform for culturally specific music alignment' claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows predictive models aligning with human auditory segmentation, providing a prior example of model-human alignment that this work extends."},{"cited_title":"Srinivasamurthy, V","cited_arxiv_id":null,"evidence_quote":"Establishes computational rhythm analysis of Indian classical music and frames the genre-specific challenge DAIRHuM targets."},{"cited_title":"Moysis, L","cited_arxiv_id":null,"evidence_quote":"Documents the difficulty of applying music deep learning to traditional music, the cultural-specificity gap DAIRHuM addresses."}],"review_version":1}