{"id":"bef03ba3-41c6-4eba-9c19-af8a5bc3f8fa","arxiv_id":"2508.19514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MQAD is a 3-million-item music QA dataset synthesized from MIR predictions and LLM-generated text, with a trained LLaMA2-Whisper model showing small captioning improvements.","lead":"This paper builds MQAD, a large music question-answering dataset with almost three million automatically generated questions and captions for over 250,000 songs. It also trains a music-language model on the dataset and reports modest captioning gains over prior pseudo-captioning baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MQAD's quality is unverified: the QA pairs are generated from automatic MIR predictions via GPT-4 with no human audit, and the in-house QA evaluation uses the same synthetic pipeline, so the core dataset claim and the claimed gains rest on an untested assumption.","rationale":"The reader's weakest assumption is precisely the point I would stress: the factuality of MQAD is never checked. My reading of the paper finds no internal evidence that softens this concern. The QA evaluation in Tables III and IV is constructed from the same MIR-output-plus-GPT pipeline as the training data, so it can reward stylistic mimicry without rewarding accurate music understanding. The captioning transfer result in Table II is real but narrow: it is a single BLEU-1 improvement of 1.21 points, METEOR is lower for the fine-tuned model, and no significance testing is reported. This does not refute the dataset's value as a pretraining corpus, but it does mean the paper's strongest external evidence is weaker than the prose suggests. The proposed human audit would directly test the load-bearing assumption. Because the concern is addressable and the dataset release could still enable independent community evaluation, I do not recommend moving away from the reader's CONDITIONAL verdict; the condition should explicitly include human validation of a QA sample and public release of the generation prompts and data.","tokens_in":8279,"tokens_out":8848,"duration_ms":83731,"concrete_test":"Conduct a human audit on a stratified random sample of 200 MQAD QA pairs spanning beat, chord, key, structure, instrument, and genre. Have at least three musicians with MIR training judge whether each answer is correct given only the 30-second audio clip, without access to the MIR-extracted event data. Report per-facet accuracy and inter-annotator agreement (e.g., Fleiss' kappa). If chord, key, or structure accuracy falls well below the reported benchmark accuracy of the underlying MIR models, or if agreement is low, the unvalidated generation chain is confirmed and the QA evaluation is measuring fidelity to a faulty synthetic ground truth.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II.A constructs the dataset by feeding automatic beat, chord, key, structure, and transcription predictions into GPT-4 Turbo to synthesize QA pairs. No human verification is reported for either the MIR predictions or the generated text. Because chord recognition, key detection, structure segmentation, and multi-instrument transcription are imperfect, systematic MIR errors propagate directly into the QA answers, and GPT-4 can introduce additional hallucinations when converting structured event lists into natural language. The QA experiments (Tables III and IV) evaluate on a test set drawn from this same generation pipeline, so high scores mainly measure how well the model matches GPT-4's question distribution and the MIR predictions, not musical correctness. The only external evidence is Table II, where the BLEU-1 gain (30.30 versus 29.09) is small, METEOR is actually lower for the fine-tuned model, and no statistical significance or repeated runs are reported. The central claim that MQAD is a large-scale factual music QA resource is therefore load-bearing on an assumption that is neither tested nor acknowledged in the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MQAD, a large-scale music question-answering and captioning dataset built on the Million Song Dataset (MSD). The dataset is constructed by running automatic MIR models (beat tracking, chord/key detection, structure segmentation, and multi-instrument transcription) on roughly 270,000 tracks and then prompting GPT-4 Turbo with the extracted features to generate nearly 3 million QA pairs and captions. The authors also present MMQAD, a multimodal LLM that combines LLaMA2-7B and Whisper, trained on MQAD. They report that MMQAD achieves state-of-the-art music captioning (e.g., BLEU-1 30.30 vs. 29.09 for LP-MusicCaps on the LP-MusicCaps-MC test set) and strong performance on an in-house QA test set. The dataset and code are publicly released.","tokens_in":8521,"tokens_out":5093,"duration_ms":44493,"significance":"If the dataset's quality and the evaluation protocol were reliable, MQAD would be a valuable community resource: it is an order of magnitude larger than existing music captioning datasets and uniquely includes time-varying structural and harmonic information. The open release of the dataset and model is a strength. However, the current manuscript does not establish that the generated QA pairs are factually correct, and the main evaluation is conducted on a test set generated by the same pipeline used to create the training data. The reported captioning gains are small, mixed across metrics, and lack statistical characterization. The paper's central claims therefore rest on assumptions that need to be tested and reported.","major_comments":[{"comment":"The dataset is built entirely from automatic MIR predictions (beat, chord, key, structure, transcription) that are then converted into QA text by GPT-4 Turbo, yet no human verification or error analysis is reported for either stage. Chord recognition, key detection, structural segmentation, and polyphonic transcription are known to be imperfect, and LLM conversion can introduce additional hallucination. Because the entire value of MQAD depends on the factual correctness of the answers, the paper must either report a human-annotation study (e.g., agreement rates on a sampled subset for each MIR category and question type) or clearly state and quantify this limitation. Without such evidence, the claim that MQAD is a high-quality factual music QA resource is not supported.","section":"II.A (Feature Extraction and QA Pair Generation)"},{"comment":"The MQAD test set (110K QA pairs) is generated by the same MIR-plus-LLM pipeline used to create the training data, so the large score gaps between MMQAD models and LP-MusicCaps on this test set largely reflect in-distribution fit to the generator's question distribution and MIR predictions, not musical understanding. The subjective evaluation in Table IV uses GPT-4 Turbo as the judge, which is the same model family that generated the QA pairs, compounding the circularity. Additionally, the description of this subjective metric (Section II.C) does not provide the prompt template or scoring rubric, making it impossible to reproduce. The authors should evaluate on a human-verified test set or at least on a subset whose ground truth has been manually checked, and they should provide full details of the LLM-as-judge protocol.","section":"III.C and Tables III–IV"},{"comment":"The headline captioning improvement is marginal: BLEU-1 increases from 29.09 (LP-MusicCaps) to 30.30 (MMQAD-C+F), while METEOR actually decreases (22.39 to 22.25), and no confidence intervals, multiple-seed variance, or statistical significance tests are reported. Given the paper's claim of state-of-the-art performance, the authors should provide error bars or significance tests across random seeds, and discuss the metric trade-offs rather than selectively highlighting BLEU-1.","section":"Table II"},{"comment":"The claim of \"state-of-the-art\" captioning is not supported by the comparison set, which includes only LP-MusicCaps and a few older baselines (Tag Concat, Template, K2C-Aug). No comparison is made to other recent music MLLMs such as MU-LLaMA or LLark under the same transfer/fine-tuning protocol. At minimum, the manuscript should restrict its claim to \"improvements over the compared baselines\" or add the missing comparisons.","section":"Abstract and Section III.B"}],"minor_comments":[{"comment":"The sentence \"The dataset and codes are at\" should be rephrased (e.g., \"The dataset and code are available at\").","section":"Abstract"},{"comment":"The description of \"Meta Questions from Music Experts\" does not specify who the experts are or how many questions were drafted; this information is needed to assess the diversity and coverage of the generated QA pairs.","section":"II.A"},{"comment":"The paper calls the GPT-4-based evaluation metric \"novel,\" but LLM-as-judge evaluation is a standard practice in NLP; the novelty claim should be toned down, and the metric should be validated against human judgments, which is currently missing.","section":"II.C"},{"comment":"The column heading \"C/A\" is ambiguous; clarify that it denotes the number of captions/answers per audio clip. Also, the text says MQAD-Full comprises 4 QA pairs and 7 captions per track, but the table does not break these out separately; a clearer breakdown would help.","section":"Table I"},{"comment":"The model name is inconsistently written as \"LLAMA2\" and \"LLaMA 2\" in different places; standardize the spelling.","section":"II.B"},{"comment":"When defining MMQAD-B, the description \"combination of LP-MusicCaps-MSD training set and MQAD training set\" should state whether the MQAD split used is the QA subset or the full dataset, since the subsequent analysis (e.g., Table III) depends on this detail.","section":"III.A"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a potentially important contribution, but the manuscript currently lacks the verification experiments that a dataset paper of this scale requires. The authors should be asked to provide human evaluation of data quality and a human-verified test set, or to substantially temper their claims to match the existing evidence. The narrow comparison set and lack of significance testing further weaken the stated state-of-the-art claim. I would not recommend rejection if the requested validation can be provided, but the revision needs to address the core validity issue rather than only cosmetic changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know about this paper: the dataset is real and large, and nothing else at this scale covers time-varying chord, key, structure, and transcription QA. That is the contribution. The evaluation attached to it is the weak part, and the paper's strongest claims rest on a test set that is generated by the exact same pipeline as the training data.\n\nWhat's good: MQAD is roughly 3 million QA items across 270k tracks from MSD. That is 5x larger than LP-MusicCaps-MSD and 500x MusicCaps, and it includes timed annotations that prior caption datasets lack. The construction pipeline is straightforward and reproducible—MIR models extract events, GPT-4 Turbo turns them into text with RAG and de-duplication. Releasing data and code for this is a solid community contribution, and the MMQAD model is a reasonable baseline even if its architecture is standard.\n\nThe soft spots matter. No human verification is reported for either the MIR features (chord/key/structure/transcription) or the generated QA text. These MIR models are imperfect, and errors in the features will propagate directly into the answers. That is not fatal for a pretraining corpus, but it means the dataset's factual accuracy is unknown. The QA evaluation in Tables III and IV is drawn from the same synthetic pipeline, so high scores mostly mean the model matches GPT-4's question distribution and the MIR outputs. The only external evidence, Table II, shows a small BLEU-1 gain (30.30 vs 29.09 over the prior best), a slightly lower METEOR, and no significance tests or repeated runs. The GPT-4 Turbo judge is also from the same model family that generated the data, which does not inspire confidence.\n\nI would not call this a takedown. The core claim—that a large, time-structured music QA dataset is useful for training music LLMs—is plausible and probably true. But the paper has not demonstrated data quality, and the evaluation section overinterprets in-distribution gains as musical understanding.\n\nThe paper deserves peer review because the resource is substantial and the community will likely use it either way. A careful referee should push for a human audit of a random subset, a cross-dataset QA evaluation, and significance testing on the captioning results. I would conditionally accept after that. I would cite it if I worked in this area, with a caveat about validation.\n\nRecommendation: send to review, with a clear request for independent evaluation.","headline":"A big, genuinely new music QA dataset whose usefulness is undercut by circular in-domain evaluation and the absence of any human audit of automatically generated answers.","tokens_in":9022,"tokens_out":1636,"would_cite":true,"duration_ms":17705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MQAD supplies nearly three million music QA pairs and shows they improve music captioning and question answering.","keywords":["music question answering","music captioning","multimodal large language model","Million Song Dataset","Music Information Retrieval","temporal music structure","GPT-4 evaluation","QA dataset"],"falsifier":"Take 500 MQAD QA pairs sampled across chord, key, structure, and transcription questions and have trained musicians check the stated facts against the audio; if the fraction of verifiably wrong answers approaches the benchmark error rates of the MIR models for chords and structure, then models trained on MQAD are being taught machine errors rather than musical facts.","tokens_in":8122,"feed_emoji":"🎵","tokens_out":5326,"duration_ms":47245,"temperature":0.7,"pith_summary":"This paper claims that a new dataset, MQAD, built on the Million Song Dataset, gives music-language models a resource they have lacked: nearly three million question-answer pairs and captions that include time-varying musical details such as chords, sections, beats, and keys. The authors argue that this scale and temporal structure let a multimodal model trained on MQAD beat prior captioning approaches on standard music captioning tests, with the best fine-tuned model reaching BLEU-1 of 30.30 against 29.09 for the strongest previous method. If true, MQAD is a training resource for music LLMs to answer structural and compositional questions rather than only tag mood or genre.","feed_headline":"3 million music QA pairs train a better music-language model","feed_subtitle":"Time-resolved chord, section, and beat questions lift captioning beyond tag-based models.","key_machinery":"The engine is a two-stage data-generation pipeline. In stage one, transformer-based Music Information Retrieval (MIR) models extract beats, chords, keys, structural sections, and multi-instrument transcriptions from each audio clip, and these events are serialized into text such as timestamped chord names and section labels. In stage two, GPT-4 Turbo, prompted with expert-authored meta questions and a retrieval-augmented generation module, converts that text into diverse QA pairs and captions. On the model side, MMQAD couples Whisper as a frozen audio encoder with LLaMA2-7B as the language backbone, trained with LoRA to keep compute costs down.","core_discovery":"The paper claims that a fully automatic pipeline can turn audio into a large, usable QA training corpus, and that training on that corpus transfers to better music captioning and question answering. Concretely, MQAD contains 4 question-answer pairs and 7 captions per track across 270,000 tracks, totaling about 3 million items; the released QA subset has about 804,000 training QA pairs. The authors report that fine-tuning on MQAD improves supervised music captioning (MMQAD-C+F: BLEU-1 30.30 vs. 29.09 for LP-MusicCaps) and that MQAD-trained models score far higher on MQAD's own QA test set (BLEU-1 51.86 vs. 15.24 for LP-MusicCaps). They also introduce a GPT-4 Turbo judge that rates model outputs across eight musical dimensions and report high scores for their models on that pseudo-subjective metric.","pith_inferences":["A human audit of a random sample of MQAD QA pairs would be the natural next test: the dataset's truthfulness as ground truth depends on the unverified correctness of the MIR extractors, and chord and structure labels are likely noisier than genre tags.","The large verbosity gap in MMQAD outputs (about 75 tokens versus 50 for LP-MusicCaps) may inflate semantic similarity scores while deflating long-gram BLEU scores, so part of the reported gain may be a style effect rather than purely factual understanding.","If the pipeline proves reliable, the same recipe of serializing MIR events and asking an LLM to generate QA pairs could transfer to other audio domains where temporal structure matters, such as sound events or speech prosody."],"forward_implications":["If MQAD's QA pairs are reliable, the music-AI community gets a way to train models that answer questions about chords, sections, and beat structure, not just genre and mood.","A model pre-trained on MQAD alone outperforms one pre-trained on the larger but caption-only LP-MusicCaps-MSD corpus, suggesting that question diversity can matter more than raw caption count for music-language pretraining.","Since MQAD-Full includes 7 captions per track beyond the released QA subset, the remaining 64% of the dataset is a ready resource for scaling captioning and text-to-music work.","The GPT-4-based subjective metric offers a cheap, repeatable proxy for evaluating music QA along multiple dimensions, which could become a standard evaluation tool."],"supporting_citations":[{"why":"Supplies the 270,000-track audio pool and the genre and tag metadata from which MQAD is built.","marker":"[16]"},{"why":"Defines the high-quality sample selection criteria that narrow MSD to roughly 20% of its tracks.","marker":"[24]"},{"why":"Provides the beat and downbeat timestamps that become the beat-format events in MQAD.","marker":"[25]"},{"why":"Provides the chord and key detection outputs that MQAD serializes into chord and key text.","marker":"[26]"},{"why":"Provides the structural-segment labels such as intro, verse, and chorus used for section questions.","marker":"[27]"},{"why":"Provides the multi-instrument transcription used for instrument and note questions.","marker":"[28]"},{"why":"Supplies the LP-MusicCaps captioning baseline and pretraining corpus that MMQAD is compared against.","marker":"[15]"},{"why":"Supplies the backbone prompt style for LLM-based QA generation, following LLark.","marker":"[14]"},{"why":"Supplies the LLaMA2-7B language backbone of the MMQAD model.","marker":"[17]"},{"why":"Supplies the Whisper audio encoder used in MMQAD.","marker":"[12]"}],"fun_headline_variants":["270k tracks, 3M music QA pairs train better captioners","Automatic music QA dataset beats tag-based captioning","3M music QA pairs from 270k tracks for LLMs","Time-resolved music QA dataset lifts captioning","MQAD: 270k songs, 3M QA pairs to train music LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole dataset rests on treating outputs of automatic MIR models as correct facts and on GPT-4-generated QA text as a faithful rendering of those facts, with no human verification of either step in the construction pipeline.","fun_headline_variants_meta":{"raw":{"variants":["270k tracks, 3M music QA pairs train better captioners","Automatic music QA dataset beats tag-based captioning","3M music QA pairs from 270k tracks for LLMs","Time-resolved music QA dataset lifts captioning","MQAD: 270k songs, 3M QA pairs to train music LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2764,"prompt_tokens":970,"completion_tokens":1794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1704}},"tokens_in":586,"tokens_out":1794,"duration_ms":12105,"temperature":1.0,"reasoning_tokens":1704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:50:56.532137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 500 MQAD QA pairs sampled across chord, key, structure, and transcription questions and have trained musicians check the stated facts against the audio; if the fraction of verifiably wrong answers approaches the benchmark error rates of the MIR models for chords and structure, then models trained on MQAD are being taught machine errors rather than musical facts.","supporting_citations":[{"cited_title":"The million song dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the 270,000-track audio pool and the genre and tag metadata from which MQAD is built."},{"cited_title":"Multi-level and multi-scale feature aggregation using pretrained convolutional neural networks for music auto-tagging,","cited_arxiv_id":null,"evidence_quote":"Defines the high-quality sample selection criteria that narrow MSD to roughly 20% of its tracks."},{"cited_title":"Modeling beats and downbeats with a time-frequency transformer,","cited_arxiv_id":null,"evidence_quote":"Provides the beat and downbeat timestamps that become the beat-format events in MQAD."},{"cited_title":"SpecTNT: A time-frequency transformer for music audio,","cited_arxiv_id":null,"evidence_quote":"Provides the chord and key detection outputs that MQAD serializes into chord and key text."},{"cited_title":"To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functions,","cited_arxiv_id":null,"evidence_quote":"Provides the structural-segment labels such as intro, verse, and chorus used for section questions."},{"cited_title":"Multitrack music transcription with a time-frequency perceiver,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-instrument transcription used for instrument and note questions."},{"cited_title":"LP-MusicCaps: LLM-based pseudo music captioning,","cited_arxiv_id":null,"evidence_quote":"Supplies the LP-MusicCaps captioning baseline and pretraining corpus that MMQAD is compared against."},{"cited_title":"Robust speech recognition via large-scale weak supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper audio encoder used in MMQAD."}],"review_version":2}