{"id":"622e69aa-003f-46a5-914c-b810a748696c","arxiv_id":"2501.00398","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Task-specific prompt ensembling with GPT-4-generated attributes and sources improves some zero-shot audio classification datasets while degrading others.","lead":"This paper introduces TSPE, a training-free method that uses GPT-4 to write task-specific audio prompts and averages their text embeddings for zero-shot audio classification. Tests on 12 datasets show inconsistent gains, and the paper's reported average improvements do not match its own results table.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I contradicts the abstract: recomputing per-dataset deltas gives mean gains of 2.30 and 2.10 for MSCLAP'23 and MSCLAP'22, not the reported 2.06 and 1.89, and several datasets show accuracy drops, so the claimed improvement range is unsupported.","rationale":"The reader's rejection is correct, but I would locate the load-bearing weakness in the arithmetic rather than solely in the manual curation. The abstract's improvement range and the averages in Section V-A are contradicted by Table I: recomputation gives means of 2.30 and 2.10, not 2.06 and 1.89, and several per-dataset deltas are negative. This is an internal inconsistency, not a matter of consensus, and it is directly checkable. The curation concern identified in the reader's weakest_assumption is also real: Sections III-B and III-D/E describe manual filtering of GPT-4 outputs using the authors' judgment about the target labels, and K=20 is selected on VocalSound in Section V-B, one of the evaluation datasets. However, that concern requires assumptions about how the prompts were constructed outside the manuscript, whereas the numerical mismatch is visible from the printed table. I therefore partially agree with the reader: same final verdict, but the most load-bearing issue is the unsupported headline statistic. If the table were corrected, the curation concern would still need to be addressed by releasing prompt artifacts or testing on held-out tasks. The paper also reports no error bars despite claiming five runs, and no code or prompts are released, so independent verification is currently impossible. None of this changes the rejection.","tokens_in":6686,"tokens_out":11918,"duration_ms":99131,"concrete_test":"Independently recompute from Table I: for each of the 24 model-dataset pairs, subtract the vanilla accuracy from the TSPE accuracy; report the mean over the 12 datasets for each model, the min and max including negative deltas, and the count of negative or zero deltas. Compare these with the abstract's '1.23-16.36%' and Section V-A's 2.06/1.89. If the recomputed means differ or any negative delta exists, the central claim is false as stated and the paper must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim fails on the paper's own table. In Table I, the MSCLAP'23 deltas are -2.11 (Beijing Opera), +6.17, +13.88, -2.02, +1.23, -1.09, +4.35, +1.70, +2.44, +5.08, 0, and -1.99; summing and dividing by 12 gives +2.30, not the 2.06 reported in Section V-A. For MSCLAP'22 the same calculation gives +2.10, not 1.89. Four of the twelve MSCLAP'23 comparisons (Beijing Opera, NSynth Instrument, Cochlscene, VocalSound) are negative and one is zero, so the abstract's statement that TSPE 'improves performance across ALMs by showing an absolute improvement of 1.23-16.36%' is not a faithful summary of Table I; the range selects only positive deltas and omits the drops. Because the central claim is quantified by these numbers, this internal inconsistency is load-bearing; without a corrected table and corrected aggregates, the improvement claim is unverified even before considering whether the manual prompt curation in Sections III-B and III-D/E generalizes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSPE (Task-Specific Prompt Ensemble), a training-free hard-prompting method for zero-shot audio classification with contrastive audio-language models such as MS-CLAP'22 and MS-CLAP'23. TSPE uses GPT-4 to generate attribute- and source-augmented prompts per task category, manually filters the prompts, and averages the text embeddings of the surviving prompts (an ensemble) before computing audio-text similarity. The authors report experiments on 12 audio classification datasets spanning five task categories, claiming an absolute improvement of 1.23–16.36% over vanilla zero-shot evaluation and average gains of 2.06% (MS-CLAP'23) and 1.89% (MS-CLAP'22). The paper also discusses hyperparameter selection of the ensemble size K=20 on the VocalSound dataset and notes some limitations, including the need for manual filtering of GPT-4 outputs.","tokens_in":7009,"tokens_out":3517,"duration_ms":32225,"significance":"If the central claim were correct, TSPE would provide a simple, training-free way to improve zero-shot audio classification, and the idea of using LLM-generated attributes and sources to enrich CLAP-style prompts is interesting and potentially useful. The paper is clearly written and evaluates on a broad set of datasets. However, the manuscript currently does not establish the claimed improvement: the reported aggregate numbers are inconsistent with the paper's own table, the improvement is not universal across datasets, and the manual, dataset-aware prompt curation together with tuning K on a test dataset undermines the zero-shot generalization claim. These issues are load-bearing, so the contribution as presented is not verified.","major_comments":[{"comment":"The aggregate results reported in §V-A are not supported by Table I. Recomputing per-dataset deltas from Table I gives an average improvement of 2.30% for MS-CLAP'23 (not 2.06%) and 2.10% for MS-CLAP'22 (not 1.89%). Moreover, the abstract's claimed range '1.23-16.36%' selects only positive deltas; for MS-CLAP'23 four of twelve comparisons are negative (e.g., Beijing Opera -2.11, NSynth Instrument -2.02, Cochlscene -1.09, VocalSound -1.99) and one is zero (SESA), while for MS-CLAP'22 seven of twelve are negative (e.g., Mridangam Stroke -3.97, GTZAN -3.21). Thus the central quantitative claim of consistent improvement is contradicted by the paper's own data.","section":"§V-A, Table I, Abstract"},{"comment":"The prompt selection procedure is manual and dataset-aware: the authors state that they 'manually map these sound attributes and sources to each task category' (§III-B) and that they manually filter 20 of 40 GPT-4-generated prompts for semantic fit (§III-D and §III-E). In addition, the ensemble size K=20 is selected by an ablation on VocalSound (§V-B), which is itself one of the twelve test datasets reported in Table I. This means the method's hyperparameters and prompt sets are chosen using the evaluation data, so the reported gains cannot be attributed to a general training-free procedure. A held-out validation split, a fully automatic prompt selection rule, or at least an ablation showing robustness to the manual curation choices is needed to support the zero-shot claim.","section":"§III-B, §III-D, §III-E, §V-B"},{"comment":"The paper claims TSPE 'significantly outperforms traditional zero-shot evaluation' (Conclusion), but no statistical significance testing is reported. All results are averages over five runs without standard deviations or confidence intervals, and many per-dataset deltas are negative while the positive average gains are modest (2.30% and 2.10% on the corrected computations). Given the high variance across datasets, the conclusion of significant improvement is not supported by the evidence presented.","section":"§V-A and Table I"}],"minor_comments":[{"comment":"The Introduction and §III-B describe prompt generation as 'automatic,' but §III-D, §III-E, and Limitation 1 make clear that manual filtering is essential; the degree of automation should be described consistently.","section":"§III-B and §VII"},{"comment":"References [2] and [3] are identical, but they are used for the Mridangam Stroke and Mridangam Tonic datasets respectively; the correct distinct sources for these datasets appear to be missing.","section":"References [2] and [3]"},{"comment":"The text introduces the abbreviation ALEs ('Audio-Language Encoders') and then uses ALMs throughout; the terminology should be unified.","section":"Introduction"},{"comment":"There are several typographical issues, including 'V ocalSound' and 'N synth' with irregular spacing, and the caption of Fig. 2 includes 'M SC LAP' with inconsistent spacing.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an arXiv preprint version. The internal inconsistency between the abstract/§V-A and Table I is a serious correctness issue that the authors must resolve before any resubmission. The manual, dataset-specific prompt curation and the tuning of K on a test dataset are more fundamental problems: they make the evaluation circular with respect to the zero-shot claim, and they are unlikely to be fully fixable without a substantial redesign of the evaluation protocol. Because these issues are load-bearing for the paper's central claim, I recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper has a real idea, but it is underevidenced in its current form. The authors apply a known technique—prompt ensembling with LLM-generated descriptors—to zero-shot audio classification, and that is a sensible extension. They also get credit for testing across 12 datasets and two CLAP checkpoints, and for including a limitations section that openly admits the manual filtering and possible bias. That is more honest than many papers.\n\nThe problem is that the central claim does not hold up on the paper's own table. Recomputing the per-dataset deltas in Table I gives mean gains of about 2.30% for MS-CLAP'23 and 2.10% for MS-CLAP'22, not the reported 2.06% and 1.89%. The abstract's \"1.23-16.36%\" range also silently drops several negative results (Beijing Opera -2.11, NSynth Instrument -2.02, Cochlscene -1.09, VocalSound -1.99, and a zero on SESA). That is not a rounding issue; it is a misrepresentation of the experiments. The paper's own text acknowledges performance decreases, so the abstract overstates.\n\nThe deeper concern is the manual curation. The authors generate 40 prompts per task with GPT-4 and then hand-pick 20 using their knowledge of the labels and datasets. That is close to fitting the prompt set to the test benchmark. The K=20 setting is tuned on VocalSound, one of the twelve test datasets. Without released prompts or code, and without a control like randomly selected prompt sets or a generic-template ensemble, there is no way to tell whether the gains come from task-specificness or from selection on the test set. The authors don't compare against a random subset of the 40 prompts, which would have been a cheap and telling baseline.\n\nIf the numbers were corrected and the method were evaluated with a held-out prompt-selection rule, this could be a useful practitioner note. As is, it is not ready for publication. A serious referee would catch the arithmetic and would likely ask for exactly the controls that are missing. My recommendation: reject in current form, but invite a revision with corrected aggregates, negative results reported, and a prompt-curation protocol that does not touch the test sets.\n\nWho is this for? People working on zero-shot audio classification or prompt engineering for CLAP-style models. It deserves referee time because the question is relevant and the experiments are broad, but the present evidence is not sufficient.","headline":"Useful idea, but the reported numbers don't survive contact with Table I and the manual curation is effectively per-dataset tuning.","tokens_in":7494,"tokens_out":1879,"would_cite":false,"duration_ms":21052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task-specific prompt ensembles improve zero-shot audio classification by 1.23–16.36% absolute over generic prompts on 12 datasets.","keywords":["zero-shot audio classification","prompt ensemble","hard prompting","audio-language model","CLAP","task-specific prompts","audio-text alignment","GPT-4 prompt generation"],"falsifier":"Run TSPE on a fresh audio dataset whose class labels are withheld from the person who selects the 20 prompts, with K fixed at 20 before evaluation; if the average gain over the generic one-line prompts vanishes, the reported improvements depended on knowing the test labels rather than on the prompt-ensemble idea.","tokens_in":6509,"feed_emoji":"🎧","tokens_out":9852,"duration_ms":91576,"temperature":0.7,"pith_summary":"This paper is trying to establish that the wording of prompts, not the model, is often the bottleneck in zero-shot audio classification. Its method, TSPE (Task-Specific Prompt Ensemble), replaces generic prompts like \"sound of a <label>\" with a short set of task-specific prompts that add sound attributes and sources, such as \"A loud sound of a jackhammer coming from a street.\" The paper reports that averaging the text embeddings of 20 such prompts per task improves zero-shot accuracy by an absolute 1.23–16.36% over vanilla prompts across 12 datasets and two audio-language models that align sound with text. If this is right, practitioners can improve zero-shot audio classification without any retraining or extra parameters. The paper also reports that a few datasets lose accuracy, so the benefit is not uniform.","feed_headline":"Prompt ensembles lift zero-shot audio accuracy by up to 16%","feed_subtitle":"Task-specific prompts outperform generic 'sound of a' templates across 12 audio datasets, with no extra training.","key_machinery":"The central mechanism is the task-specific prompt ensemble: for each class label, a set of hard prompts is written as \"A <attribute> sound of a <label>\", \"A sound of a <label> coming from a <source>\", and \"A <attribute> sound of a <label> can be heard from a <source>\". Each prompt is encoded by the text encoder, the resulting embeddings are averaged to form one class-level text representation, and classification is the cosine similarity between that averaged representation and the audio embedding. The ensemble size K=20 is selected by an ablation on VocalSound, and the prompt set itself is filtered by hand to remove GPT-4 hallucinations and semantically mismatched attribute-source pairs. The averaging is what makes the method an ensemble rather than a single prompt.","core_discovery":"The central claim, stated on the paper's own terms, is that task-specific hard prompts generated from label information improve audio-text alignment in contrastive audio-language models. The method groups class labels into five task categories, asks GPT-4 to generate 40 candidate prompts per category using sound attributes and sound sources, manually keeps 20 that fit the category, and then averages the text embeddings of those 20 prompts before computing cosine similarity with the audio embedding. On the 12 evaluation datasets, TSPE outperforms vanilla zero-shot prompting for both CLAP-type models tested, with absolute gains from 1.23% to 16.36% and average gains near 2%. The paper notes that performance decreases on some datasets, attributing the drops to prompts that are not linguistically rich enough for those tasks.","pith_inferences":["Editorial inference: an automated LLM judge could replace the manual filtering of 40 to 20 prompts; whether it reproduces the human-chosen set would directly test how much of the reported gain is curation.","Editorial inference: because K=20 was tuned on VocalSound, one of the 12 evaluation datasets, the optimal ensemble size is probably task-dependent and should be re-tuned on a held-out split for new tasks.","Editorial inference: the prompt-ensemble recipe is a natural fit for other text-conditioned audio tasks, such as audio retrieval and sound-event localization, but that transfer is not demonstrated in the paper."],"forward_implications":["A new audio classification task can be handled with no model training: one LLM call generates candidate prompts, a human filters them, and the model's existing text encoder averages the embeddings.","The same 20-prompt set for a task category can be reused across datasets in that category, such as the five instrument-recognition datasets tested.","Because the method only changes text inputs, it can compose with other zero-shot techniques and with any audio-language model that accepts natural-language prompts.","The reported gains are not universal: several datasets lose accuracy, so practitioners should validate TSPE per task before deployment."],"supporting_citations":[{"why":"It supplies the CLAP audio-language models evaluated, giving the zero-shot baselines and the audio/text encoders used in every experiment.","marker":"[19]"},{"why":"It provides the GPT-4 model used to generate the candidate sound attributes, sources, and task-specific prompts.","marker":"[13]"},{"why":"It is the VocalSound dataset, used in the ablation that sets the ensemble size K=20 and also one of the 12 evaluation datasets.","marker":"[12]"},{"why":"It supplies CLIP, the contrastive pretraining and zero-shot evaluation paradigm that TSPE extends to audio.","marker":"[16]"},{"why":"It defines hard prompting, the training-free prompt family that TSPE adopts instead of learned soft prompts.","marker":"[14]"},{"why":"It represents learnable soft prompting, the training-requiring alternative that motivates TSPE's training-free design.","marker":"[15]"}],"fun_headline_variants":["Training-free prompt tweaks boost zero-shot audio by 16%","Task-specific audio prompts beat generic templates, no training","Zero-shot audio classification improved with task-specific prompts","TSPE: tailored prompts lift zero-shot audio by 16%","Task-specific prompts boost zero-shot audio by up to 16% on 12 datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human selection of 20 prompts per task, and the choice to use 20 prompts based on a test on one of the evaluation datasets, do not amount to quietly tuning the method to the 12 benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Training-free prompt tweaks boost zero-shot audio by 16%","Task-specific audio prompts beat generic templates, no training","Zero-shot audio classification improved with task-specific prompts","TSPE: tailored prompts lift zero-shot audio by 16%","Task-specific prompts boost zero-shot audio by up to 16% on 12 datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3327,"prompt_tokens":901,"completion_tokens":2426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2337}},"tokens_in":517,"tokens_out":2426,"duration_ms":16397,"temperature":1.0,"reasoning_tokens":2337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:51:42.602765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TSPE on a fresh audio dataset whose class labels are withheld from the person who selects the 20 prompts, with K fixed at 20 before evaluation; if the average gain over the generic one-line prompts vanishes, the reported improvements depended on knowing the test labels rather than on the prompt-ensemble idea.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the GPT-4 model used to generate the candidate sound attributes, sources, and task-specific prompts."},{"cited_title":"”Wav2clip: Learning robust audio representa- tions from clip.” ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","cited_arxiv_id":null,"evidence_quote":"It supplies the CLAP audio-language models evaluated, giving the zero-shot baselines and the audio/text encoders used in every experiment."},{"cited_title":"”V ocalsound: A dataset for improving human vocal sounds recognition.” ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","cited_arxiv_id":null,"evidence_quote":"It is the VocalSound dataset, used in the ablation that sets the ensemble size K=20 and also one of the 12 evaluation datasets."},{"cited_title":"”Learning transferable visual models from natural language supervision.” International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"It supplies CLIP, the contrastive pretraining and zero-shot evaluation paradigm that TSPE extends to audio."}],"review_version":1}