{"id":"f9485eec-b5b4-4c5d-bc41-f15dcc758314","arxiv_id":"2501.13261","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-3.5 detects synthetic annotation errors in beat, chord, and key tasks above random chance, with accuracy partly influenced by musical concepts in the prompt.","lead":"This paper tests whether GPT-3.5 can catch deliberately inserted errors in beat, chord, and key annotations for MIDI music. It finds the model beats random guessing in all three tasks and that adding musical concepts to prompts often helps, though the effect is inconsistent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic-error validity is the load-bearing gap: the above-chance accuracies may not transfer to real MIR annotation errors, a limitation the paper itself defers to future work.","rationale":"The reader's weakest assumption—that synthetic errors are representative of real MIR annotation errors—is indeed the most load-bearing concern. The paper's methodology intentionally generates errors at controlled rates with random selection, but this does not simulate the structured failure modes of actual MIR systems. The paper even concedes that real-error evaluation is future work, which is an explicit statement of the limitation. This concern does not imply the experiments were done carelessly; it means the central claim is conditional on an untested external-validity assumption. A secondary observation supports the same conclusion: the abstract's accuracy figures are not the Basic Concepts numbers from Table 2 but appear to be the best per-task setting (WS 0.6520 for beat tracking, F1 0.6480 for chord under Domain Masking, and F1 0.5972 for key under '+Scale'). This makes the reported headline numbers harder to interpret but does not change the main verdict. I also note an internal inconsistency in the random baselines: for binary tasks with 30% injected errors and a 50% random-positive policy, expected precision would be approximately 0.30, whereas Table 2 reports random precision around 0.58. This suggests the baseline implementation may differ from the text, but correcting the baseline would not overturn the qualitative above-random finding; it mainly affects the size of the margin. Because the primary external-validity concern is already reflected in the reader's CONDITIONAL verdict, no verdict change is warranted; the paper should be accepted only conditionally on demonstrating that the result generalizes to real MIR errors.","tokens_in":11670,"tokens_out":5828,"duration_ms":59574,"concrete_test":"Re-run the same three error-detection tasks on the same MAPS and POP909 subsets, but replace synthetic corruptions with real MIR system outputs: use a standard beat tracker, a chord estimation system, and a key estimation algorithm to generate predictions from the same MIDI data. Feed these predictions into the exact prompts from Section 3.2 and compute the same metrics (WS for beat tracking, weighted F1 for chord and key). If GPT's performance on real system errors is not significantly above random or drops far below the Table 2 values, the synthetic-error results overstate real judging ability. To control for label distributions, report results separately for matched error rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GPT exhibits music understanding as an MIR judge rests on the assumption that the deliberately inserted errors in Section 4.1 are representative of real MIR annotation errors. In beat tracking, errors are random beat insertions (9%), deletions (12%), and offsets (9%), each generated independently of musical content. In chord extraction, 30% of root, quality, or inversion attributes are flipped at random; in key estimation, 30% of key labels are replaced by random alternative keys. Real MIR errors are not content-independent: beat trackers fail systematically at tempo changes and rubato, chord estimators confuse real chords due to voicing or non-chord tones, and key estimators are fooled by modulations and tonal ambiguity. A text-only LLM can spot a randomly flipped label by checking whether the label matches the immediately visible notes, which is a much easier task than diagnosing realistic, structurally driven failures. The paper explicitly acknowledges this in the conclusion: 'we will consider evaluating LLMs' judging ability on real MIR errors instead of synthetic ones.' Because the central quantitative claim is measured only on synthetic errors, the conclusion that GPT has solid real-world music judging ability is not yet established. Additionally, the abstract's headline accuracies (65.20%, 64.80%, 59.72%) do not all correspond to the Basic Concepts condition in Table 2; they appear to select the best-performing concept-augmentation setting per task, which further weakens the interpretation of the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper explores whether GPT-3.5 can act as a music judge for three MIR tasks by converting performances into symbolic MIDI-like JSON, injecting synthetic errors into beat, chord, and key annotations, and asking GPT to locate or judge those errors. It proposes a six-component prompt template and a concept-augmentation protocol that either adds or masks music concepts. The experiments report above-random F-scores/weighted scores on all three tasks and claim a positive correlation between concept information and detection accuracy.","tokens_in":11907,"tokens_out":7086,"duration_ms":84875,"significance":"The work is timely and useful as a training-free baseline for LLM-based MIR evaluation: it uses public datasets (MAPS and POP909), makes code available, and defines concrete error-detection metrics. If the findings are robust, the result would justify further study of LLM reasoning as a judge for symbolic MIR outputs. However, the current manuscript does not yet establish the two headline claims: the reported accuracies come from different concept settings, the concept-information trend is not consistent across tasks, and the single-run GPT evaluations are not statistically anchored. The significance is therefore conditional on revision.","major_comments":[{"comment":"The headline numbers 65.20%, 64.80%, and 59.72% are not all from the Basic Concepts condition: in Table 2 they match the '+Rhythm' row for beat tracking (WS=0.6520), the 'Domain Masking' row for chord extraction (f=0.6480), and the '+Scale' row for key estimation (f=0.5972). The text does not disclose this selection. Please either report a single consistent condition for the headline, or explicitly state that these are best per-task settings with the Basic Concepts results reported in the same table.","section":"Abstract, Section 4.3, Table 2"},{"comment":"The claim of a positive correlation between detection accuracy and the amount of concept information is not supported by Table 2. In chord extraction, the F-score increases from 0.6207 in Basic Concepts to 0.6376 under Music Attribute Masking, 0.6362 under Task Masking, and 0.6480 under Domain Masking, the opposite of the predicted trend. In key estimation, Basic, +Scale, and Task Masking are almost indistinguishable (0.5965, 0.5972, 0.5960). A monotonic or ordinal relation should be tested, or the claim should be restricted to the beat-tracking task, where the ordering is more consistent.","section":"Section 4.3, Table 2"},{"comment":"The GPT results are single-run and no temperature or model version is reported, even though the paper itself notes \"high randomness in output, sensitivity to prompts, and hallucination.\" Without repeated sampling or confidence intervals, it is unclear whether the observed advantages over the random baseline (e.g., key estimation Basic F-score 0.5965 vs. random 0.5186) are statistically reliable. Please provide repeated trials, report variance, and perform a significance test or at least a conservative interval comparison.","section":"Section 4.3"},{"comment":"The evaluation is carried out exclusively on synthetic, content-independent label errors (random beat insertions/deletions/offsets, random chord-attribute flips, random key replacements). Real MIR errors are typically systematic and context-dependent, so above-chance detection of random flips may overstate judging ability for realistic system outputs. Since the conclusion explicitly defers testing on \"real MIR errors instead of synthetic ones,\" the abstract's \"solid ground\" claim should be softened, or the paper should include at least one experiment on real MIR outputs.","section":"Section 4.1, Section 5"}],"minor_comments":[{"comment":"\"Aconcept augmentation\" should read \"A concept augmentation.\"","section":"Section 1, first paragraph"},{"comment":"\"quality of theet chord\" contains a typo; it should be \"quality of the chord.\"","section":"Figure 1, chord prompt"},{"comment":"The abstract uses the word \"accuracy,\" but the evaluation reports F-scores and the weighted WS score; please align the terminology throughout.","section":"Abstract and Section 4.2"},{"comment":"The random baseline for chord and key estimation is described as \"detects incorrectness with a probability of 50%,\" but the table reports weighted precision, recall, and F1; a short derivation or explanation of how the tabulated baseline values follow from this rule would help readers.","section":"Section 4.3, random baselines"}],"recommendation":"major_revision","confidential_remarks":"This is a reasonable ISMIR-scale empirical study with a useful dataset/code contribution, but the headline claims need to be reframed and statistically grounded. The concept-correlation claim is contradicted by the authors' own Table 2, and the reported accuracies are not drawn from a single consistent condition. I do not see grounds for rejection if the authors are willing to revise these points and add repeated trials; the current version is not yet publishable as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know about this paper? It is the first benchmark I know of that uses a text-only LLM (GPT-3.5) as a judge for MIR annotation errors on symbolic inputs, and it shows above-random detection of deliberately inserted errors in beat tracking, chord extraction, and key estimation. The idea is timely and the execution is honest, but the headline numbers are less robust than the abstract suggests.\n\nWhat is genuinely useful: the public datasets (MAPS, POP909), the code repo, the clear prompt structure, and especially the concept-augmentation strategy—masking music concepts to see how performance changes is a nice way to probe what the model is actually using. The case study is also illustrative. The authors are upfront in the conclusion that real MIR errors remain future work.\n\nWhere it gets soft. First, the synthetic errors are content-independent random flips: random beat insertions/deletions/offsets, random chord attribute changes, random key substitutions. Real MIR errors are systematic—beat trackers fail at tempo changes, chord estimators confuse voicings, key estimators struggle with modulation. Detecting a randomly flipped label is a much easier problem, so the accuracies are likely an upper bound on real judging ability. The paper admits this. Second, the abstract cites 65.20%, 64.80%, and 59.72%, but those correspond to different prompt conditions per task (the best setting in each table), not to a single uniform configuration. That is misleading. Third, there are no repeated trials or temperature settings, even though Section 4.3 mentions high output randomness. The random baselines have confidence intervals; the GPT numbers are single point estimates, so we can't tell if the beat and key improvements are real. Fourth, the claimed positive correlation with concept information doesn't hold up consistently: chord extraction performs best under full domain masking, and key results are flat across conditions.\n\nProportionally, these are fixable flaws, not fatal ones. The paper doesn't overclaim its real-world applicability, and the negative result on concept info is at least reported in the tables. For an MIR audience, this is a legitimate exploratory result that sets a baseline. I would send it to peer review—the topic is timely and the methodology is transparent enough that a revision with repeated trials, single-configuration numbers, and ideally real MIR errors would make it solid. I'd read it as evidence that GPT has some symbolic music pattern-matching ability, not as a proven judge for production MIR workflows.","headline":"A promising but fragile first benchmark for LLMs as MIR annotation judges; the synthetic-error setup and single-run numbers mean the headline accuracies should be read as upper bounds, not robust evidence of music reasoning.","tokens_in":12426,"tokens_out":3756,"would_cite":true,"duration_ms":33786,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A text-only LLM can judge music annotations: GPT detects errors in beat tracking, chord extraction, and key estimation at rates above random when music is converted to symbolic MIDI and fed through structured prompts.","keywords":["LLM as judge","music information retrieval","symbolic music","prompt engineering","error detection","beat tracking","chord extraction","key estimation"],"falsifier":"Use the same prompts and metrics but replace the synthetic corruptions with annotations from real beat trackers, chord extractors, and key estimators on the same MAPS and POP909 data. If GPT's detection accuracy on these real outputs falls to the random baseline levels reported in Table 2, the paper's claim that GPT exhibits measurable music understanding would be refuted.","tokens_in":11476,"feed_emoji":"🎵","tokens_out":6570,"duration_ms":59384,"temperature":0.7,"pith_summary":"This paper asks whether a text-only large language model can judge music information retrieval output without any music-specific training. It converts music into symbolic MIDI, deliberately inserts errors into beat, chord, and key annotations, and asks GPT to spot those errors. GPT exceeds random baselines in all three tasks, at 65.20% for beat tracking, 64.80% for chord extraction, and 59.72% for key estimation. The authors also show that adding musical concepts to the prompt tends to improve detection, while masking those concepts changes or degrades performance. This matters because it suggests a training-free route to coupling language-model cognition with music analysis systems.","feed_headline":"GPT out-judges random guessing on music annotations","feed_subtitle":"Text-only GPT, fed symbolic MIDI and structured prompts, catches wrong beats, chords, and keys above chance.","key_machinery":"The central machinery is a six-component prompt: background and task definition, a description of the relevant music concepts, an explanation of the symbolic input format, step-by-step reasoning techniques, an output format specification, and the data itself. Music is encoded as MIDI notes and annotation labels in JSON, which unifies music and text without adapters or fine-tuning. A concept augmentation scheme adds or masks concepts at the music-attribute, task, and domain levels to test whether GPT's judgments track the musical content of the prompt.","core_discovery":"The paper reports that GPT, given only symbolic MIDI and textual prompts, identifies deliberately corrupted MIR annotations at rates above a random baseline in all three tasks: 65.20% for beat tracking, 64.80% for chord extraction, and 59.72% for key estimation. It also finds that GPT's error-finding performance broadly rises with the amount of musical concept information the prompt provides, while masking all music concepts degrades performance. The authors interpret these results as evidence that text-only reasoning carries a measurable amount of music understanding and can serve as a training-free judge for MIR tasks.","pith_inferences":["Editorial inference: Because the injected errors are synthetic, the reported accuracies are likely an optimistic estimate for real MIR errors; real errors can be subtler and less uniformly distributed.","Editorial inference: The concept-augmentation results could be used as a diagnostic for whether an LLM actually uses music concepts or just responds to wording; one could vary concept names while holding underlying structure constant.","Editorial inference: A production system could combine this judge with a perception network, using LLM explanations as weak labels to improve MIR models without human annotation.","Editorial inference: The observed trade-off in beat tracking, where adding a rhythm concept raised correct passes on true positives but lowered detection of false positives and negatives, suggests concept prompts may bias the judge toward specific error types."],"forward_implications":["A text-only model can serve as a training-free baseline judge for symbolic MIR tasks, giving future work concrete numbers to beat.","Prompt design is a substantive variable: explicit music concepts such as rhythm, chroma, and scale measurably shift GPT's detection performance.","Full audio-to-text alignment is not required for a first-pass music judge; symbolic input plus prompt engineering is sufficient for above-chance performance.","The same six-component prompt and controlled-error protocol can be applied to other symbolic MIR tasks without retraining the model.","Concept masking provides a way to observe what happens when music-specific language is removed, isolating the contribution of music knowledge from generic sequence reasoning."],"supporting_citations":[{"why":"Shows an LLM answers music theory questions above random, the motivation for testing GPT as a music judge.","marker":"[1]"},{"why":"Provides the chain-of-thought prompting technique that the prompt structure uses.","marker":"[5]"},{"why":"Supplies the beat tracking evaluation framework (TP, FP, FN) and the 70 ms tolerance used to define offset errors.","marker":"[18]"},{"why":"Defines the chord label syntax used in the chord error detection task.","marker":"[19]"},{"why":"Supplies the MAPS classical piano MIDI recordings used for beat tracking tests.","marker":"[20]"},{"why":"Provides the metrical annotations aligned to MAPS used to define ground-truth beats.","marker":"[21]"},{"why":"Supplies the POP909 pop MIDI dataset used for chord and key error detection.","marker":"[22]"}],"fun_headline_variants":["GPT beats chance at spotting bad music annotations","Text-only GPT finds music annotation errors above random","GPT's music ear: catches annotation errors above chance","GPT judges music annotations: better than random"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the artificially inserted beat, chord, and key errors at controlled rates are representative of real MIR annotation errors, so that detecting them is a valid measure of music understanding.","fun_headline_variants_meta":{"raw":{"variants":["GPT beats chance at spotting bad music annotations","Text-only GPT finds music annotation errors above random","GPT's music ear: catches annotation errors above chance","GPT judges music annotations: better than random"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2939,"prompt_tokens":851,"completion_tokens":2088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":2030}},"tokens_in":467,"tokens_out":2088,"duration_ms":14589,"temperature":1.0,"reasoning_tokens":2030,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:18:23.787110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the same prompts and metrics but replace the synthetic corruptions with annotations from real beat trackers, chord extractors, and key estimators on the same MAPS and POP909 data. If GPT's detection accuracy on these real outputs falls to the random baseline levels reported in Table 2, the paper's claim that GPT exhibits measurable music understanding would be refuted.","supporting_citations":[{"cited_title":"ENSTDkCl","cited_arxiv_id":null,"evidence_quote":"Supplies the POP909 pop MIDI dataset used for chord and key error detection."},{"cited_title":"Exploring GPT's Ability as a Judge in Music Understanding","cited_arxiv_id":"2501.13261","evidence_quote":"Shows an LLM answers music theory questions above random, the motivation for testing GPT as a music judge."},{"cited_title":"In the computer music domain, the research to combine text and audio LLMs is also popular","cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting technique that the prompt structure uses."},{"cited_title":"Count the occurrences of each pitch class (C, C#, D, etc.) and determine their relative frequencies","cited_arxiv_id":null,"evidence_quote":"Supplies the beat tracking evaluation framework (TP, FP, FN) and the 70 ms tolerance used to define offset errors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the chord label syntax used in the chord error detection task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MAPS classical piano MIDI recordings used for beat tracking tests."},{"cited_title":"correctness","cited_arxiv_id":null,"evidence_quote":"Provides the metrical annotations aligned to MAPS used to define ground-truth beats."}],"review_version":1}