{"id":"0cb06151-4d1d-4709-ab15-b2107e252465","arxiv_id":"2607.18718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On a new 3000-item audio-only benchmark stripped of text-solvable questions, the best AI system scores 58.33%, random guessing scores 25.46%, and all 36 competing systems fail the same 233 items.","lead":"DCASE 2026 launched a listening quiz for AI: 3000 multiple-choice questions filtered so that they cannot be answered from text alone. On this audio-only test the best of 36 systems scored 58.33% (random guess: 25.46%), and every system missed the same 233 questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADF's 'more than one silent model' retention rule lets single-model text-solvable items into ADQA-Bench, so audio-dependence is not established.","rationale":"The paper is a challenge summary; the leaderboard facts are internally consistent and the authors appropriately flag confounds (dev/eval incomparability, sub-pp ties, paradigm/backbone confounding). Those strengths make the report credible as a record. However, the scientific payoff — that ADQA-Bench measures audio understanding — depends on ADF having removed all text-solvable items. The Hard-ADF '>1 model' threshold is not a minor implementation detail: it is a logical mismatch with §2.1. Allowing exactly one text-capable model to solve an item means the property definition is not satisfied. The 25.46% random floor corroborates that artifacts leak. If a text-only probe exceeds chance, the top-accuracy and failure-set interpretations weaken; if it does not, the concern is resolved. This supports keeping the reader's CONDITIONAL verdict: accept the factual report but require the filter-validation test before treating the numbers as evidence about audio understanding. The reader's weakest assumption already points to filter completeness; this pass sharpens it to the specific retention threshold, so agreement is 'agree.'","tokens_in":11894,"tokens_out":5463,"duration_ms":54066,"concrete_test":"Have the organizers run a strong text-only LLM (e.g., Llama-3-70B or GPT-4.1) on the 3000 ADQA-Bench items with question and options but no audio, then compare accuracy to the 25.46% random floor. Stratify by Hard-ADF outcome: items with 0, 1, and >1 silent-audio correct (the last should be absent by construction). If text-only accuracy on the 0- and 1-correct strata is significantly above chance (e.g., >30% with a binomial test), the ADF retention rule is too permissive and the benchmark contains text-solvable items, invalidating the 'audio-dependence' claim. As a control, repeat on the 1607-item development set, which uses the same filter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"§2.1 defines an audio-dependent item as one whose 'correct answer cannot be recovered from the text alone.' The only mechanism enforcing this is the ADF cascade (§2.2). Yet Hard-ADF discards an item only if 'more than one blind model still picks the correct option.' By that rule, an item that exactly one silent-audio LALM can solve from text is deliberately kept, directly contradicting the §2.1 definition. The Soft-ADF perplexity check drops only when the correct option has the lowest perplexity; a strong second-place textual cue survives. The LLM commonsense check prunes only items a single text LLM happens to answer correctly. Human verification checks answer correctness and distractor plausibility, not text-solvability. So the benchmark's defining property is not 'cannot be recovered from text alone' but 'not caught by three coarse filters.' The paper's own random-guess floor of 25.46% (§2.4), attributed to option-length and distractor artifacts, is direct evidence that non-audio signals remain in the final set. Since the headline numbers (58.33% top accuracy, 11.91 pp dev-to-eval drop, 233 universal failures) are all interpreted as measuring audio understanding, the central claim rests on an unvalidated filter-completeness assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is the official summary of DCASE 2026 Task 5 on Audio-Dependent Question Answering. It introduces ADQA-Bench, a 3,000-item evaluation set produced by an Audio-Dependency Filtering (ADF) cascade (silent-audio probing, per-option perplexity, an LLM commonsense check, and human verification), and reports results from 14 teams and 36 submissions across two parameter tracks. The main findings are: a Chung-Ang University ensemble (Lim_CAU_4) reaches 58.33% top accuracy; the lightweight champion scores 57.30%; the average dev-to-eval gap across 30 comparable submissions is -11.91 pp; MOSS-Audio-8B-Thinking is the most common backbone; and 233 evaluation items are missed by all systems. The paper also surveys methodological trends (LoRA, SFT, GRPO/GDPO, prompt/voting ensembles) and explicitly flags several limitations, including dev-only baselines, sub-percentage-point tie levels, confounded paradigm/backbone comparisons, and the absence of an ablation for the winning system's acoustic tagger.","tokens_in":12067,"tokens_out":8442,"duration_ms":74894,"significance":"If the ADF filtering is sound, the paper is a valuable community resource: a public benchmark whose scoring is designed to penalize textual-prior guessing, plus a first-year snapshot of system capabilities. The paper deserves credit for repeatedly flagging its own inferential limits (dev-only baselines, sub-%-point ties, confounded comparisons, missing ablation) and for making the challenge results transparent. The 233 universally-failed items and the dev-to-eval gap are concrete, falsifiable measurements that should inform future benchmark design. The main caveat — the filter-completeness assumption — is discussed below; if not addressed, the benchmark's headline interpretation as measuring 'audio understanding' is not established.","major_comments":[{"comment":"§2.1 defines an audio-dependent item as one whose correct answer 'cannot be recovered from the text alone.' The ADF cascade (§2.2) does not enforce this: Hard-ADF drops an item only if *more than one* silent-audio LALM picks it, so an item that exactly one blind model can solve from text is retained. Soft-ADF drops only when the correct option has lowest perplexity; a second-place textual cue survives. The LLM filter prunes only what one text-only LLM answers correctly; human review checks answer correctness and distractor plausibility, not text-solvability. The 25.46% random-guess floor in §2.4 (option-length/distractor artifacts) is in-paper evidence that non-audio signals remain. Since the abstract and Section 5 interpret the top accuracy, dev-to-eval drop, and 233 universal failures as measuring audio understanding, the central claim rests on an unvalidated filter-completeness assump","section":"§2.1–§2.2"},{"comment":"§5.2 reports a mean dev-to-eval drop of -11.91 pp and attributes it to the evaluation set being 'designed to be harder than the development split.' The two splits are different item sets, so the gap conflates item-set difficulty with development-set overfitting and sampling variability. The paper acknowledges overfitting for some teams and n=30, but the causal attribution to a harder eval set is not supported by a controlled comparison. Provide construction details or a matched-difficulty analysis (e.g., same items in both conditions, or a difficulty-calibration table); otherwise limit the claim to the observed drop.","section":"§5.2"}],"minor_comments":[{"comment":"The 'Mean (%)' for Qwen3-Omni-30B is computed over five submissions from only two teams; the text acknowledges this, but the table itself is unannotated. Add an asterisk or a note so the mean is not over-read.","section":"Table 3"},{"comment":"The 25.46% random-guess level is computed on the development set. The statement 'Every system beats the random-guess baseline' should clarify that eval-set scores are being compared with a dev-set estimate, not a matched eval-set baseline.","section":"§2.4 / Abstract"},{"comment":"The rule that 'gaps of about a percentage point should be read as ties' would benefit from a binomial confidence-interval justification for n=3000; as written it is a reasonable rule of thumb but presented as a fixed threshold.","section":"§3"},{"comment":"The caption mentions a dashed line marking the 49.65% mean, but the line is not labeled in the plot itself. Add a legend entry.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The ADF completeness issue in §2.2 is the key risk to the benchmark's central claim. The authors are otherwise transparent; if they can provide a residual-leakage validation or scale back the audio-dependence claims, I would support acceptance. The challenge summary itself is well within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest and useful challenge summary, but don't take the benchmark's audio-dependence guarantee at face value. The new numbers are real: the 3000-item ADQA-Bench, the 36-submission leaderboard with 58.33% top, the mean 11.91 pp dev-to-eval drop, and the 233 items every system missed. The paper is candid in the right places—it says organizer baselines are dev-only and not comparable to eval, that ~1 pp gaps are ties, that paradigm comparisons are confounded with backbone choice, and that the winning acoustic tagger is not ablated. That level of disclosure is above average for a challenge report.\n\nThe soft spot is the ADF pipeline that defines the benchmark. Hard-ADF keeps an item if only one silent-audio LALM happens to pick the correct option, and the perplexity and LLM checks only catch the strongest textual cues. The paper's own random-guess floor of 25.46%—above 25% due to option-length and distractor artifacts—is direct evidence that non-audio signals survive into the final set. So the claim that ADQA-Bench items are audio-dependent is stronger than the filter actually supports. The paper never runs a text-only probe on the final 3000 items, and it interprets the 233 universal failures as shared capability gaps without auditing for label noise. Both are fixable and the discussion hints at the need, but the analysis stops short.\n\nFor a challenge summary this is solid work. The leaderboard facts are internally consistent and the honesty about confounds makes it trustworthy as a report. The interpretation—that 58.33% measures genuine audio understanding—is conditional on filter completeness. A serious referee should ask for a text-only probe on ADQA-Bench and a re-audit of the universal-failure set before the benchmark becomes a standard instrument. I'd take it to peer review, and I'd cite it for the leaderboard and the 233-item failure set, but I'd hedge any strong claim about audio dependence until the validation is done.","headline":"Honest challenge summary whose benchmark's audio-dependence guarantee is weaker than advertised—useful new numbers, but filter completeness needs validation.","tokens_in":12797,"tokens_out":2916,"would_cite":true,"duration_ms":27520,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A filtered 3,000-question audio benchmark shows the best model answers only 58.3% correctly, meaning the task measures genuine audio understanding, not textual shortcuts.","keywords":["audio-dependent question answering","benchmark filtering","large audio-language models","textual priors","audio understanding","multiple-choice evaluation","DCASE challenge","audio reasoning"],"falsifier":"Take a random sample of ADQA-Bench evaluation items and run a strong text-only LLM (with no audio input) on the question and options, measuring its accuracy against the 25.46% random-guess baseline. If the text-only model scores significantly above that baseline on the same items, it demonstrates that some benchmark items are still solvable from text alone, falsifying the claim that all 3,000 items are audio-dependent. Similarly, removing option-length and distractor artifacts should bring the random-guess baseline to exactly 25.00%; if it does not, textual priors remain in the benchmark.","tokens_in":11624,"feed_emoji":"🎧","tokens_out":2258,"duration_ms":28348,"temperature":0.7,"pith_summary":"This paper introduces ADQA-Bench, a 3,000-item multiple-choice audio benchmark whose items are explicitly filtered to remove questions answerable from text alone. The authors' claim is that the filtering pipeline (silent-audio probing, perplexity checks, LLM commonsense review, human verification) leaves only audio-dependent questions, so the measured accuracy of current large audio-language models reflects true audio understanding. The top system reaches 58.33% accuracy, well above the 25.46% random-guess baseline, and all 36 submissions miss the same 233 items, suggesting shared capability gaps. The paper also documents a consistent drop of about 12 percentage points from development to evaluation sets, indicating the hidden evaluation split is harder by design.","feed_headline":"Best model answers only 58% of audio-dependent questions","feed_subtitle":"A new 3,000-item benchmark filters out text-solvable items; all 36 submissions fail the same 233 questions.","key_machinery":"The key mechanism is the Audio-Dependency Filtering (ADF) pipeline, a four-stage cascade that keeps only questions whose correct answer depends on the audio signal. First, Hard-ADF replaces audio with silence and discards items if more than one blind LALM selects the correct option. Second, Soft-ADF drops items where the correct option has the lowest language-model perplexity under silent audio. Third, a text-only LLM commonsense filter prunes questions answerable from world knowledge. Fourth, a two-pass human review verifies that the labeled answer is correct given the audio and that distractors are plausible but acoustically wrong. This cascade is the load-bearing component that defines wh","core_discovery":"The central discovery is that a carefully filtered audio question-answering benchmark can separate audio-driven performance from linguistic priors. The Audio-Dependency Filtering (ADF) cascade removes items that silent-audio LALMs, perplexity-based text preferences, or a text-only LLM can solve; the surviving 3,000 items are claimed to be audio-dependent. On this benchmark, the best system—an ensemble of MOSS-Audio-8B-Thinking and Qwen3-Omni-30B—achieves 58.33% accuracy, while the sub-10B lightweight winner scores 57.30%, only 1.03 percentage points behind. Every submission misses a common set of 233 items (7.8% of the evaluation set), and an oracle that picks the best answer across all syst","pith_inferences":["A direct test of the filter's completeness would be to run a strong text-only language model on ADQA-Bench items; if it scores significantly above the 25.46% random-guess baseline, then some text-solvable items still pass the ADF cascade, and the benchmark's audio-dependence claim would need qualification.","The residual 'option-length and distractor artifacts' that push random-guess accuracy to 25.46% are in-paper evidence that the filter does not fully eliminate text-only signals; a stronger filter could target these artifacts directly.","Extending ADQA-Bench to open-ended generation, as the paper suggests, would require an audio-blind judge; if such a judge could be built, it would likely reveal even larger performance gaps than multiple-choice accuracy shows.","The shared failure on 233 items may point toward a specific audio property (e.g., fine-grained temporal or event-level cues) that current encoders under-represent; probing those items with targeted acoustic augmentations could isolate the missing capability."],"forward_implications":["If the ADF filtering is sound, ADQA-Bench provides a reliable measure of audio-grounded reasoning, and the 58.33% top accuracy indicates substantial headroom for improving large audio-language models.","The consistent 11.91 percentage-point average drop from development to evaluation confirms that the hidden split is harder, making it a more reliable test of generalization than the development set.","The 233 universally-failed items suggest a shared capability gap among current backbones—likely stemming from common audio encoders or pre-training strategies—rather than random errors.","The small 1.03 percentage-point gap between an 8-billion-parameter model and a 96-billion-parameter ensemble implies that architectural choices, data curation, and inference-time ensembling can be as important as raw scale.","The finding that CoT-targeted fine-tuning helps on some backbones but not others, and that many reasoning-time interventions fail, indicates that audio-answer alignment is not yet a solved problem."],"fun_headline_variants":["Text-proof audio quiz: best AI scores 58%","New audio QA benchmark stumps top model at 58%","Audio-only questions: even best AI fails 42%","Benchmark filters text clues; best audio AI hits 58%","All 36 audio QA models flunk same 233 questions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the ADF cascade catches every text-solvable item—that is, any item that defeats all three automated filters and passes human review is genuinely audio-dependent, so if a text-solvable item slips through all filters, it is incorrectly counted as audio-dependent.","fun_headline_variants_meta":{"raw":{"variants":["Text-proof audio quiz: best AI scores 58%","New audio QA benchmark stumps top model at 58%","Audio-only questions: even best AI fails 42%","Benchmark filters text clues; best audio AI hits 58%","All 36 audio QA models flunk same 233 questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1186,"prompt_tokens":904,"completion_tokens":282,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":648,"tokens_out":282,"duration_ms":5882,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:33:23.686401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ADQA-Bench evaluation items and run a strong text-only LLM (with no audio input) on the question and options, measuring its accuracy against the 25.46% random-guess baseline. If the text-only model scores significantly above that baseline on the same items, it demonstrates that some benchmark items are still solvable from text alone, falsifying the claim that all 3,000 items are audio-dependent. Similarly, removing option-length and distractor artifacts should bring the random-guess baseline to exactly 25.00%; if it does not, textual priors remain in the benchmark.","supporting_citations":[],"review_version":1}