Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

On-demand benchmarks from your own scores test whether MLLMs actually hear and read music

2026-07-08 19:24 UTC pith:ZHK7U54A

load-bearing objection Useful on-demand multimodal music-perception MCQ framework from user scores; the claim that text-only + white-noise prove genuine perception is under-supported for notation and symbolic paths. the 3 major comments →

arxiv 2607.06015 v1 pith:ZHK7U54A submitted 2026-07-07 cs.SD

Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

Tom\'a\v{s} Sourada , Katia Vendrame , Jan Haji\v{c} jr This is my paper
classification cs.SD
keywords music perceptionmultimodal large language modelsbenchmarkingsymbolic musicMusicXMLmultiple-choice questionscross-modal evaluationon-demand benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large static music benchmarks for multimodal language models are expensive to run, hard to trust on music outside their fixed set, and often can be solved without ever perceiving the music itself. This paper introduces MusICA-MetaBench, a framework that turns user-provided symbolic scores (such as MusicXML) into multiple-choice questions automatically, using fixed templates drawn from music pedagogy. The same questions can be asked in audio, notation-image, or symbolic-file form, so models can be compared across modalities on the same musical material. Text-only and white-noise controls are used to check that correct answers require genuine perception of the musical input rather than language priors or residual metadata. On a demonstration set of Bach-style chorales (ChoraleBricks), the authors also find how many questions are needed for statistically reliable model rankings. If the method holds, anyone with a score library can generate a focused, perception-centered benchmark for the music they actually care about.

Core claim

MusICA-MetaBench automatically builds on-demand multimodal multiple-choice benchmarks from user-supplied symbolic music. Pedagogy-aligned question templates applied to structured encodings produce questions that, when paired with text-only and white-noise baselines, measure music perception rather than non-perceptual shortcuts, and experimentally chosen sizes support statistically reliable model comparisons (shown on ChoraleBricks).

What carries the argument

Automatic template-based generation of multiple-choice perception questions from symbolic encodings (e.g., MusicXML), rendered into audio, notation images, and symbolic files, with text-only and white-noise baselines that isolate genuine cross-modal music perception from language priors and residual cues.

Load-bearing premise

That fixed pedagogy-style question templates on structured symbolic scores, plus text-only and white-noise checks, are enough to guarantee that correct answers need real perception of the music rather than leftover metadata, template artifacts, or language shortcuts that those two controls miss.

What would settle it

On a held-out set of user scores, a strong multimodal model that scores near chance on both the text-only and white-noise controls still scores far above chance on the rendered audio/notation questions without using any musical content—e.g., by exploiting OCR artifacts, spectrogram shortcuts, or template regularities that survive the baselines.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Anyone with a MusicXML (or similar) library can generate a custom perception benchmark without building a new static dataset.
  • Model rankings can be compared across audio, notation images, and symbolic files on identical musical material.
  • Benchmark size can be chosen so that pairwise model differences are statistically reliable for a given score collection.
  • Claims of “music understanding” can be stress-tested against text-only and white-noise controls before they are accepted as perception.
  • Evaluation cost scales with the user’s own data rather than with ever-larger fixed public suites.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If symbolic-to-multimodal rendering is faithful, the same pipeline could stress-test whether models that claim score following actually track voice leading, cadence, or key rather than surface pattern matching.
  • The method may transfer to other structured cultural media (e.g., dance notation or chess scores) where templates plus modality controls can separate perception from language priors.
  • Failure modes that survive white-noise and text-only baselines—such as OCR of printed accidentals or spectrogram texture cues—would become the next natural control targets.
  • Community score libraries could become shared “what we care about” testbeds, shifting evaluation from fixed corpora toward domain-specific perception stress tests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces MusICA-MetaBench, a meta-benchmark framework that automatically derives on-demand multimodal multiple-choice benchmarks from user-provided symbolic music (e.g., MusicXML) via fixed pedagogy-aligned question templates. Generated items are intended to probe music perception competencies across three modalities—audio, notation images, and symbolic files—and to support systematic cross-modal comparison. The authors demonstrate the pipeline on the ChoraleBricks dataset, experimentally choose benchmark sizes argued to yield statistically reliable model comparisons, and report text-only and white-noise baseline controls as evidence that the questions measure music perception rather than non-perceptual shortcuts. The central claim is that this dataset-specific paradigm overcomes three limitations of static music benchmarks: evaluation cost, unclear transfer beyond the fixed corpus, and lack of cross-modal assessment.

Significance. If the framework and controls hold, the contribution is practically significant for multimodal music evaluation: on-demand, user-data-driven benchmarking reduces the need for large static suites and enables modality-matched comparisons on music the community actually cares about. Pedagogy-aligned templates and explicit attention to sample-size reliability are strengths relative to ad-hoc music LLM leaderboards. The work is constructive and falsifiable in principle (baseline gaps, size experiments). Significance of the headline claim that the questions “do measure music perception,” however, rests on whether the reported controls close residual shortcut paths in every modality; that is the load-bearing empirical hinge for the paper’s positioning against prior “music understanding” benchmarks.

major comments (3)
  1. The abstract’s assertion that text-only and white-noise baselines show the questions “do measure music perception” is load-bearing for the paper’s second claimed advance (benchmarks that require perception). White-noise is an audio-domain control and does not address notation-image or symbolic-file pathways. For MusicXML/symbolic input, template questions on pitch, key, harmony, or rhythm can be solved by structured parsing of tags without auditory or visual music perception, yet still beat text-only and remain untouched by white-noise. For notation images, OCR-plus-language reasoning can succeed without musical perception. Even for audio, white-noise only shows that non-noise acoustic input helps; it does not show use of piece-specific musical content versus generic audio statistics or option–template correlations that appear only when real audio is present. Content-mismatch, scrambled-
  2. The claim that experimentally determined benchmark sizes “ensure statistically reliable model comparisons” on ChoraleBricks is central to the efficiency argument. The manuscript must report the design of that size experiment (power analysis or resampling procedure, effect-size assumptions, multiple-comparison handling across models/modalities/templates, and the decision rule for N). If reliability is only shown for a subset of templates or one modality, the cross-modal comparison claim should be narrowed accordingly. Tables or figures of pairwise separation vs. N, with uncertainty, are needed so readers can judge whether the chosen sizes actually support the stated comparisons.
  3. “Pedagogy-aligned” competencies are part of the framing that distinguishes MusICA from generic music QA. The paper should specify which pedagogical sources or skill taxonomies the templates implement, how coverage was validated (expert review, curriculum mapping), and whether template difficulty or construct validity was checked beyond face validity. If alignment is only author-asserted, the claim should be softened to “pedagogy-inspired templates” unless supporting evidence is added.
minor comments (5)
  1. Clarify early (abstract/intro) the exact symbolic formats supported at generation time versus at evaluation time, and whether MIDI is treated as first-class input or only as a render path from MusicXML.
  2. Define “music perception” operationally versus “music understanding” when contrasting prior benchmarks, so the baseline design can be judged against a stated construct.
  3. When reporting baseline gaps, give absolute accuracies and chance levels per modality and template family, not only relative improvements, so shortcut residual performance is visible.
  4. State licensing and redistribution constraints for user-provided music and for ChoraleBricks-derived items, since on-demand generation from third-party scores raises reuse questions for shared leaderboards.
  5. If code and template catalogs are released, document the exact generation seed, option-shuffling policy, and any exclusion rules for degenerate items (e.g., trivial keys, empty voices) so results are reproducible.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and constructive report. The three major comments correctly identify load-bearing points in our positioning: (i) what the text-only and white-noise controls actually establish about “music perception,” (ii) the transparency of the sample-size experiment that underwrites reliable comparisons on ChoraleBricks, and (iii) the strength of the “pedagogy-aligned” framing. We agree that the abstract and related claims over-reach relative to the controls and evidence currently reported, and we will revise the manuscript accordingly: soften and modality-qualify the perception claim, fully document the size experiment (design, decision rule, and separation-vs-N evidence), and either ground or soften the pedagogy language. Below we respond point by point. We believe these revisions address the recommendation for major revision without changing the core technical contribution of the meta-benchmark pipeline.

read point-by-point responses

  1. Referee: The abstract’s assertion that text-only and white-noise baselines show the questions “do measure music perception” is load-bearing. White-noise is audio-only and does not address notation-image or symbolic pathways. Symbolic MusicXML items can be solved by structured tag parsing without perception; notation images by OCR-plus-language reasoning. Even for audio, white-noise only shows non-noise input helps, not piece-specific musical content vs. generic audio statistics or option–template correlations. Content-mismatch / scrambled controls are needed; residual shortcut paths remain.

    Authors: We agree that the abstract’s unqualified claim is stronger than the controls support, and that white-noise is an audio-domain control only. Text-only is a modality-agnostic “no music input” baseline; white-noise only probes whether non-noise audio helps. Neither rules out (a) MusicXML tag parsing for symbolic input, (b) OCR-plus-linguistic reasoning for notation images, nor (c) residual audio shortcuts (generic acoustics, option–template correlations that appear only with real audio). We will revise the abstract and main text to state only what the controls establish: that performance drops without music-bearing input (text-only) and, for audio, without structured acoustic content (white-noise), and that residual non-perceptual pathways remain possible in each modality. We will add an explicit limitations subsection on shortcut risks (symbolic parsing, OCR, audio statistics) and, where feasible within revision, report content-mismatch / scrambled-content or modality-scrambled controls on a subset of templates to tighten the claim. We will not claim that the present baselines “show the questions do measure music perception” across all modalities. revision: yes

  2. Referee: The claim that experimentally determined benchmark sizes “ensure statistically reliable model comparisons” on ChoraleBricks is central. The manuscript must report the design of that size experiment (power analysis or resampling procedure, effect-size assumptions, multiple-comparison handling across models/modalities/templates, and the decision rule for N). If reliability is only shown for a subset of templates or one modality, the cross-modal comparison claim should be narrowed. Tables or figures of pairwise separation vs. N, with uncertainty, are needed.

    Authors: We agree that the size claim is central to the efficiency argument and that the current manuscript does not report the experiment design at the level required for readers to judge reliability. In revision we will fully document: (1) the procedure used (resampling / bootstrap of item subsets and pairwise model separation as a function of N), (2) the effect-size and separation criteria that drove the decision rule for chosen N, (3) how we handled multiplicity across models, modalities, and templates (or explicitly note if we did not correct and why), and (4) whether the chosen N was validated for all templates and modalities or only a subset. We will add tables and/or figures of pairwise separation versus N with uncertainty bands so that the chosen sizes can be audited. If reliability is only demonstrated for a subset of templates or modalities, we will narrow the cross-modal comparison claim accordingly rather than assert global statistical reliability. revision: yes

  3. Referee: “Pedagogy-aligned” competencies distinguish MusICA from generic music QA. The paper should specify which pedagogical sources or skill taxonomies the templates implement, how coverage was validated (expert review, curriculum mapping), and whether template difficulty or construct validity was checked beyond face validity. If alignment is only author-asserted, soften to “pedagogy-inspired templates” unless supporting evidence is added.

    Authors: The referee is correct that “pedagogy-aligned” currently rests on author design of templates around standard ear-training / theory competencies (pitch, key, harmony, rhythm, etc.) rather than a documented mapping to a named curriculum or taxonomy with external validation. We did not conduct formal expert review, curriculum mapping, or psychometric checks of difficulty/construct validity beyond face validity of the templates. We will therefore either (a) add a short subsection naming the pedagogical sources that informed the template set and describing any informal expert feedback we can document, or—if that evidence remains thin—(b) soften the language throughout (abstract, introduction, and claims) to “pedagogy-inspired” templates that target common music-perception competencies, and state clearly that construct validity and curriculum coverage are not formally validated. We will not retain the stronger “aligned with music pedagogy” phrasing without supporting evidence. revision: yes

standing simulated objections not resolved
  • We cannot fully close residual non-perceptual shortcut paths in every modality (MusicXML structured parsing, notation OCR-plus-language reasoning, and audio generic-statistics / option–template correlations) with the present control suite alone; even with added content-mismatch/scrambled controls on a subset, a complete multi-modality shortcut audit is beyond what a single revision can guarantee, so the perception claim will remain carefully qualified.

Circularity Check

0 steps flagged

No significant circularity: constructive on-demand benchmarking framework with external baselines, not a self-reducing derivation.

full rationale

MusICA-MetaBench is a constructive evaluation system: fixed pedagogy-aligned templates applied to user-provided symbolic music (e.g., MusicXML) generate multimodal MCQs, with text-only and white-noise baselines and experimental sizing for statistical reliability on ChoraleBricks. None of the load-bearing claims reduce by construction to their inputs in the sense of the circularity taxonomy. The assertion that questions “do measure music perception” is an empirical claim supported by control conditions, not a fitted parameter renamed as a prediction, nor a uniqueness theorem or ansatz imported from the authors’ prior work, nor a renaming of a known empirical law. Template design that encodes the competencies it aims to probe is ordinary benchmark construction, not self-definitional circularity of a claimed first-principles result. Residual validity concerns (whether white-noise/text-only suffice for notation/symbolic modalities, OCR or structured-parse shortcuts) are correctness/design risks, not circular reductions. No quoted step exhibits Eq. X = Eq. Y by construction or a self-citation chain that forces the central result. Score 0 with empty steps is therefore the warranted finding.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

This is an evaluation-methods paper, not a physical theory. The main free choice is the empirically selected benchmark size for reliable comparisons. Load-bearing axioms are domain assumptions: that pedagogy-aligned templates on symbolic music operationalize perception, that the named baselines suffice to isolate perception, and that symbolic encodings ground matched multimodal items. No new physical entities are postulated; MusICA-MetaBench is a named software framework, not an invented ontological object.

free parameters (1)
  • benchmark size N for reliable comparisons = not stated in abstract
    Authors experimentally determine quiz sizes that ensure statistically reliable model comparisons for the ChoraleBricks setup; N is chosen from experimental outcomes rather than derived a priori.
axioms (3)
  • domain assumption Music perception competencies can be operationalized as multiple-choice items generated from symbolic music via fixed pedagogy-aligned templates.
    Core design premise of MusICA-MetaBench; if false, auto-generated items may not measure the intended skills.
  • domain assumption Text-only and white-noise baselines are sufficient to establish that performance requires music perception.
    Abstract relies on these two controls to claim the questions measure perception; other non-perceptual shortcuts may remain.
  • domain assumption Structured symbolic encodings (e.g., MusicXML) faithfully ground the same musical facts across audio, notation-image, and symbolic modalities.
    Cross-modal comparison assumes consistent ground truth under each rendering.

pith-pipeline@v0.9.1-grok · 6390 in / 2508 out tokens · 60665 ms · 2026-07-08T19:24:49.487942+00:00 · methodology

0 comments
read the original abstract

Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure "music understanding" often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · 13 internal anchors

  1. [1]

    Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

    INTRODUCTION Music exists digitally across three modalities: audio recordings, symbolic encodings (e.g., MIDI, MusicXML, ABC notation), and sheet music images. Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in audio and visual understanding [1–3], © T. Sourada, K. Vendrame, and J. Hajiˇc jr.. Licensed under a Creative Commo...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    A known weakness of closed QA is that some bench- marks can be partially solved without perceiving any musical content, through reasoning over answer options alone [5,6]

    RELATED WORK Recent benchmarks for music understanding (2024–2025) predominantly evaluate (M)LLMs via question answering (QA) [5, 7, 9–11, 13, 14, 22], with multiple-choice (closed QA) being the dominant format. A known weakness of closed QA is that some bench- marks can be partially solved without perceiving any musical content, through reasoning over an...

    work page 2024

  3. [3]

    analysis

    METHODOLOGY MusICA-MetaBench is organized around a predefined set of question templates (Tab.1), each with an implemented function that extracts the ground truth and distractor op- tions from aMusicXML. The resulting question-option pairs apply across symbolic files, sheet images, and audio, provided the dataset contains piece-aligned data in those format...

    work page 2026

  4. [4]

    DATASET: CHORALEBRICKS We use the ChoraleBricks dataset [20] as input for MusICA-MetaBench for most of the experiments. It in- cludes multitrack recordings of ten chorales, each with so- prano, alto, tenor, and bass parts played by a variety of wind instruments, along with sheet music in MusicXML and PNG, ideal for MusICA-MetaBench. For generating a MusIC...

    work page

  5. [5]

    Select- ing an appropriate benchmark size requires balancing sta- tistical power against evaluation cost

    CALIBRATION OF BENCHMARK SIZE The full ChoraleBricks-derived benchmark would con- tain 12,666 items (4,222×3 modalities), making ex- haustive evaluation computationally impractical. Select- ing an appropriate benchmark size requires balancing sta- tistical power against evaluation cost. Because MusICA- MetaBench targets real-world model selection rather t...

    work page 2025

  6. [6]

    not really listening

    IS PERCEPTION REQUIRED? To validate our distractor generation, we test whether an- swering benchmark questions requires perception of the musical content; that the benchmark does not suffer from the “not really listening” problem [5], extended to “not looking at the image” and “not reading the symbolic score” in our multimodal setup). We compare MLLM perf...

    work page

  7. [7]

    Bassus Ch1-F4

    BENCHMARK TEST-DRIVE We demonstrate MusICA-MetaBench in practice on the ChoraleBricks and ChoraleSynth datasets. 7.1 Model Comparison Results on ChoraleBricks We set the minimum effect size of interest to 6 % and consider three modalities: audio recording, sheet image Model Acc. (%) Time Price ($) Gemini 3.1 Pro 59.0 12 h 58.4 Qwen3 Omni 30B 46.3 2 h N/A ...

    work page 2026

  8. [8]

    try to guess the correct answer

    CONCLUSIONS AND FUTURE WORK MusICA-MetaBench is a programmatic pipeline that gen- erates QA benchmarks from user-provided musical data, targeting musical perception across modalities (audio, sheet images, symbolic formats) and scalable to optimal size. It offers an alternative to large static benchmarks, avoiding data leakage and generality constraints. W...

    work page 2026

  9. [9]

    Human-centred AI for a Sustainable and Adaptive Society

    ACKNOWLEDGMENTS This work was supported by the project “Human-centred AI for a Sustainable and Adaptive Society” (reg. no.: CZ.02.01.01/00/23_025/0008691), co-funded by the Eu- ropean Union, and partially by the SVV project number 260 821. The work described herein has been using ser- vices provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https:...

    work page

  10. [10]

    First, as coding assistants during development of the MusICA-MetaBench pipeline, including code drafting, de- bugging, and refactoring

    AI USAGE STATEMENT Large language models were used in three capacities in this work. First, as coding assistants during development of the MusICA-MetaBench pipeline, including code drafting, de- bugging, and refactoring. Second, as writing assistants for drafting and rewriting portions of this manuscript. All AI- assisted outputs were critically reviewed,...

    work page

  11. [11]

    ETHICS STATEMENT Data and copyright.MusICA-MetaBench is demon- strated on the ChoraleBricks and ChoralSynth datasets, both of which are publicly available research datasets dis- tributed under open licences. A key motivation of our framework is precisely to avoid the copyright concerns in- herent in large static benchmarks: by enabling researchers to gene...

    work page

  12. [12]

    GPT-4o System Card

    OpenAI, :, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. M ˛ adry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Car- ney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A....

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasu- pat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, L. Marris, S. Petulla, C. Gaffney, A. Aharoni, N. Lintz, T. C. Pais, H. Jacobsson, I. Szpektor, N.-J. Jiang, K. Haridasan, A. Omran, N. Saunshi, D. Bahri, G. Mishra, E. Chu, T. Boyd, B. Hekman, A. Parisi, C. Zhang, ..., and W. Helmholz,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Qwen3-Omni Technical Report

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhu, Y . Lv, Y . Wang, D. Guo, H. Wang, L. Ma, P. Zhang, X. Zhang, H. Hao, Z. Guo, B. Yang, B. Zhang, Z. Ma, X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, M. Yang, D. Liu, X. Ren, B. Zheng, R. Men, F. Zhou, B. Yu, J. Yang, L. Yu, J. Zhou, and J. Lin, “Qwen3-Omni Technical Report,” Sep....

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Cmi- bench: A comprehensive benchmark for evaluating music instruction following,

    Y . Ma, S. Li, J. Yu, E. Benetos, and A. Maezawa, “Cmi- bench: A comprehensive benchmark for evaluating music instruction following,” inProceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, South Korea, September 21-25, 2025, J. Nam, D. Jeong, K. Choi, L. Su, M. Fuentes, T. Nakano, X. Hu, and H. H. ...

    work page doi:10.5281/zenodo.17811400 2025

  16. [16]

    Are you really listening? Boost- ing Perceptual Awareness in Music-QA Bench- marks,

    Y . Zang, S. O’Brien, T. Berg-Kirkpatrick, J. McAuley, and Z. Novack, “Are you really listening? Boost- ing Perceptual Awareness in Music-QA Bench- marks,”Proceedings of the 26th International Preprint – July 8, 2026 Society for Music Information Retrieval Confer- ence (ISMIR 2025), Sep. 2025. [Online]. Available: https://ismir2025program.ismir.net/poster...

    work page 2026

  17. [17]

    MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

    X. Yue, T. Zheng, Y . Ni, Y . Wang, K. Zhang, S. Tong, Y . Sun, B. Yu, G. Zhang, H. Sun, Y . Su, W. Chen, and G. Neubig, “MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark,” May 2025, arXiv:2409.02813 [cs]. [Online]. Available: http://arxiv.org/abs/2409.02813

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

    B. Weck, I. Manco, E. Benetos, E. Quinton, G. Fazekas, and D. Bogdanov, “MuChoMusic: Eval- uating Music Understanding in Multimodal Audio- Language Models,” Aug. 2024, arXiv:2408.01337 [cs]. [Online]. Available: http://arxiv.org/abs/2408.01337

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    HumMusQA: A Human-written Music Understanding QA Benchmark Dataset,

    B. Weck, P. Puentes, A. Poltronieri, S. Prabhu, and D. Bogdanov, “HumMusQA: A Human-written Music Understanding QA Benchmark Dataset,” in Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026), E. V . Epure, S. Oramas, S. Doh, P. Ramoneda, A. Kruspe, and M. Sordo, Eds. Rabat, Morocco: Association for Computational Linguistics, Mar. 202...

    work page 2026

  20. [20]

    WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning,

    G. Mundada, Y . Vishe, A. Namburi, X. Xu, Z. Novack, J. McAuley, and J. Wu, “WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning,” Sep. 2025, arXiv:2509.04744 [cs]. [Online]. Available: http://arxiv.org/abs/2509.04744

    work page arXiv 2025

  21. [21]

    MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

    J. Chen, W. Ma, P. Liu, W. Wang, T. Song, M. Li, C. Wang, J. Qin, R. Zhang, and C. Chen, “MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models,” Aug. 2025, arXiv:2506.23009 [cs]. [Online]. Available: http: //arxiv.org/abs/2506.23009

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following,

    J. Zhao, Y . Li, W. Li, and K. Yoshii, “ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following,” Sep. 2025, arXiv:2509.23350 [cs]. [Online]. Available: http://arxiv.org/abs/2509.23350

    work page arXiv 2025

  23. [23]

    ChatMusician: Understanding and Generating Music Intrinsically with LLM

    R. Yuan, H. Lin, Y . Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y . Wu, C. Liu, Z. Zhou, Z. Ma, L. Xue, Z. Wang, Q. Liu, T. Zheng, Y . Li, Y . Ma, Y . Liang, X. Chi, R. Liu, Z. Wang, P. Li, J. Wu, C. Lin, Q. Liu, T. Jiang, W. Huang, W. Chen, E. Benetos, J. Fu, G. Xia, R. Dannenberg, W. Xue, S. Kang, and Y . Guo, “ChatMusician: Understanding and Generating M...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning,

    Z. Wang, Z. Yang, Y . Luo, Y . Li, X. Qu, Z. Qiao, H. Zhang, R. Zhan, D. F. Wong, J. Zhou, and Y . Cheng, “Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning,” Sep. 2025, arXiv:2509.04059 [cs]. [Online]. Available: http://arxiv.org/abs/2509.04059

    work page arXiv 2025

  25. [25]

    Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores

    C. Dai, Y . Yang, K. Li, H. Zhou, S. Liang, Z. Bo, E. Liu, G. Jin, H. An, H. Zhang, P. Jing, K. Lee, Z. Zhang, X. Li, and M. Sun, “Musical Score Understanding Benchmark: Evaluating Large Lan- guage Models’ Comprehension of Complete Musical Scores,” Nov. 2025, arXiv:2511.20697 [cs]. [Online]. Available: http://arxiv.org/abs/2511.20697

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio,

    Q. Zhao, Y . Cao, Y . Huang, H. Y . Leong, F. Zhang, K.-H. Yap, and W. Hu, “MuseAgent-1: Interactive Grounded Multimodal Understanding of Music Scores and Performance Audio,” Jan. 2026, arXiv:2601.11968 [cs]. [Online]. Available: http://arxiv.org/abs/2601. 11968

    work page arXiv 2026

  27. [27]

    Music21: A toolkit for computer-aided musicology and symbolic music data,

    M. S. Cuthbert and C. Ariza, “Music21: A toolkit for computer-aided musicology and symbolic music data,” inProceedings of the 11th International Society for Music Information Retrieval Conference, ISMIR 2010, Utrecht, Netherlands, August 9-13, 2010, J. S. Downie and R. C. Veltkamp, Eds. International Society for Music Information Retrieval, 2010, pp. 637–...

    work page 2010

  28. [28]

    Statistical universals reveal the structures and functions of human music,

    P. E. Savage, S. Brown, E. Sakai, and T. E. Currie, “Statistical universals reveal the structures and functions of human music,”Proceedings of the National Academy of Sciences, vol. 112, no. 29, pp. 8987–8992, 2015. [Online]. Available: https: //www.pnas.org/doi/abs/10.1073/pnas.1414495112

    work page doi:10.1073/pnas.1414495112 2015

  29. [29]

    Conver- gent evolution in a large cross-cultural database of mu- sical scales,

    J. M. McBride, S. Passmore, and T. Tlusty, “Conver- gent evolution in a large cross-cultural database of mu- sical scales,”PLOS ONE, vol. 18, no. 12, p. e0284851, Dec. 2023. [Online]. Available: https://journals.plos. org/plosone/article?id=10.1371/journal.pone.0284851

    work page doi:10.1371/journal.pone.0284851 2023

  30. [30]

    Integer Ratio Priors on Musical Rhythm Revealed Cross-culturally by Iterated Reproduction,

    N. Jacoby and J. H. McDermott, “Integer Ratio Priors on Musical Rhythm Revealed Cross-culturally by Iterated Reproduction,”Current Biology, vol. 27, no. 3, pp. 359–370, Feb. 2017. [Online]. Avail- able: https://www.cell.com/current-biology/abstract/ S0960-9822(16)31511-1

    work page 2017

  31. [31]

    ChoraleBricks: A Modular Multitrack Dataset for Wind Music Research,

    S. Balke, A. Berndt, and M. Müller, “ChoraleBricks: A Modular Multitrack Dataset for Wind Music Research,”Transactions of the International Society for Music Information Retrieval (TISMIR), Feb

    work page

  32. [32]

    Available: https://transactions.ismir

    [Online]. Available: https://transactions.ismir. net/articles/10.5334/tismir.252

    work page doi:10.5334/tismir.252

  33. [33]

    ChoralSynth: Synthetic Dataset of Choral Singing

    J. Narang, V . D. L. Vega, X. Lizarraga, O. Mayor, H. Parra, J. Janer, and X. Serra, “ChoralSynth: Synthetic Dataset of Choral Singing,” Nov. 2023, arXiv:2311.08350 [cs]. [Online]. Available: http: //arxiv.org/abs/2311.08350 Preprint – July 8, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Jamendo- QA: A Large-Scale Music Question Answering Dataset,

    J. Koh, S. Y . Kim, Y . Choi, and G. H. Choi, “Jamendo- QA: A Large-Scale Music Question Answering Dataset,” Sep. 2025, arXiv:2509.15662 [cs]. [Online]. Available: http://arxiv.org/abs/2509.15662

    work page arXiv 2025

  35. [35]

    Assessing Factual Music Comprehension in Large Audio Language Models

    D. C. Lin, M. Freeman, and J. Thickstun, “Factual and Musical Evaluation Metrics for Music Language Models,” Nov. 2025, arXiv:2511.05550 [cs]. [Online]. Available: http://arxiv.org/abs/2511.05550

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen, “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” Jun. 2024, arXiv:2311.16502 [cs]. [Online]. Available: ht...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  37. [37]

    Teaching LLMs Music Theory with In-Context Learning and Chain-of-Thought Prompting: Pedagogical Strategies for Machines

    L. Pond and I. Fujinaga, “Teaching LLMs Music The- ory with In-Context Learning and Chain-of-Thought Prompting: Pedagogical Strategies for Machines,” Mar. 2025, arXiv:2503.22853 [cs]. [Online]. Avail- able: http://arxiv.org/abs/2503.22853

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Sloboda,The Musical Mind: The Cognitive Psychology of Music, ser

    J. Sloboda,The Musical Mind: The Cognitive Psychology of Music, ser. Oxford psychology series. Clarendon Press, 1985. [Online]. Available: https: //books.google.cz/books?id=oZCfAAAAMAAJ

    work page 1985

  39. [39]

    Bloom, D

    B. Bloom, D. Krathwohl, and B. Masia,Taxonomy of Educational Objectives: The Classification of Educational Goals, ser. Taxonomy of Educational Objectives: The Classification of Educational Goals. D. McKay, 1956, no. v. 1. [Online]. Available: https://books.google.cz/books?id=hos6AAAAIAAJ

    work page 1956

  40. [40]

    The kodály method and its musical basis,

    L. Dobszay, “The kodály method and its musical basis,”Studia Musicologica Academiae Scientiarum Hungaricae, vol. 14, no. 1/4, pp. 15–33, 1972. [On- line]. Available: http://www.jstor.org/stable/901863

    work page 1972

  41. [41]

    Hearing the Order: Investigating Position Bias in Large Audio-Language Models,

    Y .-X. Lin, C.-A. Li, S.-L. Wei, P.-C. Chen, H.-H. Chen, and H.-y. Lee, “Hearing the Order: Investigating Position Bias in Large Audio-Language Models,” Feb. 2026, arXiv:2510.00628 [cs]. [Online]. Available: http://arxiv.org/abs/2510.00628

    work page arXiv 2026

  42. [42]

    An Analysis of Variance Test for Normality (Complete Samples),

    S. S. Shapiro and M. B. Wilk, “An Analysis of Variance Test for Normality (Complete Samples),”Biometrika, vol. 52, no. 3/4, pp. 591–611, 1965. [Online]. Available: https://www.jstor.org/stable/2333709

    work page arXiv 1965