{"id":"30a01e5d-15e3-4ee6-b53f-2e7c52cc23c8","arxiv_id":"2505.03054","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BLAB is the first benchmark to evaluate audio language models on hour-scale audio, and state-of-the-art models score below 22% exact match on most of its tasks.","lead":"BLAB is a new benchmark that plays hour-long audio clips to AI models and asks them to locate words, count speakers, estimate durations, and judge emotions. Current AI models, including Gemini 2.0 Pro and GPT-4o, score very low, showing that they cannot handle long-form audio.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gemini's word-localization F1 is capped by its 8,096-token output limit (~261 timestamps vs ~10,500 ground-truth words), so near-zero word F1 and the long-vs-short gap partly reflect decoding capacity, not long-audio understanding.","rationale":"The reader's concern about unverified Claude-3 NE labels is valid and should be checked, but it affects only one localization task and is fixable by human verification. The word-localization output-token cap is more load-bearing because it is an internal property of the evaluation protocol: the model physically cannot emit timestamps for all ~10,500 words, so the task's F1 ceiling is near 5% regardless of audio understanding. This also contaminates the paper's strongest duration comparison (Figure 3a), since shortened clips fit within the output budget. The broader conclusion that long-form audio understanding is weak is still supported by other tasks, such as advertisement localization (frame-F1 4.93), speaker number estimation (EMA 8.50), and event duration (EMA up to 4.95), so rejection is not warranted. However, the benchmark should be revised to either use a ceiling-aware metric, allow compressed or chunked output, or drop word localization from the headline duration analysis. The abstract should also be corrected to say that six models were evaluated on BLAB/BLAB-MINI, with only Gemini models on the full-length tasks. These are concrete, necessary conditions rather than fatal flaws, so conditional acceptance is the appropriate verdict.","tokens_in":24744,"tokens_out":9458,"duration_ms":110984,"concrete_test":"Run a ceiling oracle for word localization: for each long BLAB word-localization sample, output the ground-truth start/end timestamps for the first K words that fit in an 8,096-token JSON response, and compute word F1. If the oracle F1 is 4-5%, the reported Gemini scores are largely explained by the decoder length cap. As a complementary check, re-run Gemini with a token-efficient protocol, e.g., chunking the hour-long audio into segments whose outputs fit the budget and then merging timestamps, or prompting for timestamps only against a provided transcript. If chunked/merged inference raises word F1 well above 1.12/0.19, word localization cannot support the duration-decline claim and should either be re-scored with a ceiling-aware metric or excluded from the headline analysis.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that audio LMs struggle with BLAB and that performance declines with duration leans heavily on the localization tasks, and word localization is the largest localization task (191 hours). In Section 4.1, the authors state that Gemini's output context of 8,096 tokens lets it emit only ~261 word timestamps per sample, about 2% of the ~10,500 ground-truth words. This creates a hard ceiling: even a perfect model that localizes the first 261 words exactly has recall 261/10,500 ~ 2.5% and F1 ~ 4.9%, only slightly above the reported 1.12 (Flash) and 0.19 (Pro). The Figure 3 comparison with BLAB-MINI is therefore not a clean duration experiment for this task: 30-second clips contain roughly 200-300 words, which fit inside the output budget, so the improvement from ~1-2 F1 to 30.22/8.61 F1 is partly an output-capacity artifact rather than evidence about long-context audio understanding. The authors disclose the cap, but the reported word F1 is still treated as a diagnostic of long-form ability. Separately, only Gemini models were run on full-length BLAB, so the abstract's claim that six models were evaluated on BLAB is not supported by the long-form results; the other four models were tested only on BLAB-MINI. The unverified Claude-3 NE labels are a real annotation concern, but the word-localization output cap is more load-bearing because it undercuts the cleanest visible instance of the duration-decline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BLAB, a benchmark of 833+ hours of long-form audio (samples from 15 minutes to 2 hours) with eight tasks across four categories: localization, counting, emotion, and duration estimation. The authors evaluate Gemini 2.0 Flash and Gemini 2.0 Pro on the full-length BLAB tasks and six audio LMs on a 30-second subset called BLAB-MINI. They report uniformly low performance on long-form tasks, with localization F1 scores near zero, exact-match accuracies below 10% on counting and duration tasks, and better but still limited performance on emotion reasoning. The paper includes analyses of duration effects, chain-of-thought prompting, and robustness to silence and Gaussian noise. The central empirical claim is that current audio LMs, including Gemini 2.0 Pro and GPT-4o, struggle with long-form audio and that performance declines with increasing duration.","tokens_in":25075,"tokens_out":8129,"duration_ms":77645,"significance":"If validated, BLAB fills a real gap in audio evaluation: most existing benchmarks cap audio at 30 seconds to 5 minutes, and timestamp localization over hour-scale audio is largely absent from prior benchmarks. The dataset construction from permissively licensed YouTube sources, the human-assisted filtering pipeline, and the planned public release of data, code, and model generations are concrete strengths. The paper also makes falsifiable observations—near-zero localization scores, systematic underestimation of speaker counts, duration overestimation, and degraded robustness on long noisy inputs—that are useful for future work on long-context audio models. The significance is conditional, however, on resolving the evaluation-coverage, metric-confounding, and annotation-validation issues detailed below.","major_comments":[{"comment":"The paper itself discloses that Gemini's 8,096-token output context limits generation to roughly 261 word timestamps per sample, against ~10,500 ground-truth words. Under the word-level F1 metric, a model that emits exactly the first 261 correct timestamps has recall 261/10,500 ≈ 2.5% and F1 ≈ 4.9%, which is only slightly above the reported 1.12 (Flash) and 0.19 (Pro). The sharp improvement on BLAB-MINI (word F1 30.22 and 8.61) is therefore partly a decoding-capacity artifact because 30-second clips contain only ~200–300 words and fit inside the output budget; the comparison does not cleanly measure duration effects. I ask the authors to either evaluate word localization on a matched output budget (e.g., only the first 261 ground-truth words), report recall@K or precision conditioned on emitted tokens, or otherwise separate output-capacity limits from long-context understanding.","section":"Sec. 4.1, Fig. 3(a), Table 2"},{"comment":"The paper claims to evaluate six audio LMs on BLAB, but full-length BLAB results (Table 2) are only for Gemini 2.0 Flash and Gemini 2.0 Pro. Qwen2-Audio, Audio Flamingo 2, Phi-4-Multimodal, and GPT-4o were evaluated only on BLAB-MINI, the ≤30-second subset. Thus the statement that 'all of them, including advanced models such as Gemini 2.0 Pro and GPT-4o, struggle with the tasks in BLAB' does not follow from the reported experiments. Please revise the abstract and conclusion to distinguish full-length evaluation (Gemini only) from short-clip evaluation (six models), or extend the evaluation to the additional models on representative long-form samples within their input limits.","section":"Abstract; Sec. 3.1 and Table 3"},{"comment":"Ground-truth entity spans for NE localization are produced by feeding WhisperX transcripts to Claude-3 and mapping extracted spans back to timestamps, with no described human verification step, unlike advertisement localization, where an annotator reviewed all predicted segments and removed false positives. Because the Limitations section acknowledges that WhisperX is not robust to overlapping speech, incomplete or misaligned entity spans would depress frame-level F1 and inflate the benchmark's difficulty. I request a validation study of the NE extraction step (e.g., manual review of a sample with correction rates, or inter-annotator agreement), and reporting of how many predicted spans were corrected.","section":"Sec. 2.1, Named Entity Localization; Limitations"},{"comment":"The size of the emotion tasks is internally inconsistent. Table 1's caption says each task has 200 question-audio-answer pairs, and Section 3.2 says emotion tasks result in 200 unique questions, but Section 2.3 reports 44 questions for Emotion Reasoning and 156 verified segments for Valence & Arousal Ranking. The reported emotion accuracies (e.g., 64.29% for Emotion Reasoning on Gemini 2.0 Pro) are therefore based on a much smaller sample than the benchmark's headline size, and no confidence intervals are given. Please correct the inconsistency and report uncertainty or per-sample variance for these small tasks.","section":"Sec. 2.3, Table 1, Sec. 3.2"}],"minor_comments":[{"comment":"In the Valence & Arousal Ranking prompt, the option list contains two entries labeled '(D)'—'Very Unpleasant' and 'Unpleasant'—and skips a unique label for one of them; fix the option letters.","section":"Sec. 3.2, Table 6"},{"comment":"The sentence 'Gemini 2.0 flash achieves a precision score of 24.37 indicating that the model predictions are correct approximately 24.58% of the time' contains two different numbers for the same precision value; clarify which is correct.","section":"Sec. 4.1"},{"comment":"The sentence 'Our goal is to measure the models ability to model's ability to disregard background noise and focus on meaningful content' contains a duplicated phrase; rewrite it.","section":"Sec. 5, robustness"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a useful contribution and the core observation that Gemini models perform poorly on long-form audio is likely correct, but the abstract oversells the evaluation coverage and the word-localization metric confounds output capacity with long-context ability. The requested fixes are within scope for a revision; I would not recommend rejection. I would also ask the authors to be careful with the small emotion-task sample sizes when drawing comparative conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BLAB is the first audio benchmark that pushes evaluation out to hour-scale clips, and it shows convincingly that current models—including Gemini 2.0 Pro—struggle with this regime. That finding alone makes it worth a serious look. The data collection is substantial: 833+ hours across eight tasks, with human verification for the word, advertisement, speaker-count, and duration annotations. The localization tasks fill a real gap; no prior benchmark asked models to localize words, named entities, and ads in long audio.\n\nThe paper is honest in places. It discloses that Gemini's 8,096-token output limit lets it emit only ~261 word timestamps per sample, roughly 2% of the ~10,500 ground-truth words. That disclosure matters, because it means the near-zero word-F1 numbers are not purely about audio understanding—they are also a decoding-capacity ceiling. The short-vs-long comparison for word localization is especially confounded: 30-second clips fit inside the output budget, so the jump from F1 ~1 to ~30 partly reflects removing the output cap, not just the duration change. This doesn't sink the paper—the decline persists in other tasks like NE and ad localization, speaker counting, and event duration—but it should be analyzed and discussed head-on rather than presented as a straight duration effect.\n\nThe bigger wording problem is the abstract and conclusion claiming six audio LMs were evaluated on BLAB. Only the two Gemini models got the full-length clips; the other four ran on BLAB-MINI, a 30-second subset. That is a real overstatement and should be fixed.\n\nTwo smaller issues. The named-entity localization labels come from Claude-3 extracting entities from WhisperX transcripts, with no human verification, unlike the other localization tasks. If those spans are incomplete, frame-level F1 is depressed and the task looks harder than it is. The emotion tasks are also tiny: 44 reasoning questions and 156 rating segments, so conclusions there are limited.\n\nOn balance, the central gap is real and the paper's infrastructure is useful. The citation pattern is fine, and the benchmark construction is independent of the models tested. I would send this to review, expecting major revisions: correct the evaluation-scope language, re-express the word-localization results with the output-cap ceiling in mind, and verify a sample of the NE labels.","headline":"BLAB is a genuinely useful first hour-scale audio benchmark, but its headline claims overstate the evaluation scope and one of its key metrics is partly an output-capacity artifact.","tokens_in":25650,"tokens_out":3102,"would_cite":true,"duration_ms":30964,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BLAB, an 833-hour benchmark of 15-minute-to-2-hour audio, finds that current audio language models, including Gemini 2.0 Pro and GPT-4o, score near floor on localization, counting, duration, and emotion tasks, with accuracy falling as…","keywords":["BLAB","long-form audio understanding","audio language models","benchmark","localization","duration estimation","speaker counting","emotion reasoning"],"falsifier":"Take a random subset of BLAB's named-entity localization items, have independent human annotators mark every entity span directly in the audio, and recompute frame-level F1 against both the human labels and the Claude-3/WhisperX labels; if models score substantially higher against human labels, or if the machine labels contain many spans humans cannot hear, then the reported NE-localization difficulty is partly an artifact of label generation rather than model limitation.","tokens_in":24589,"feed_emoji":"🎧","tokens_out":8139,"duration_ms":79566,"temperature":0.7,"pith_summary":"BLAB is a benchmark built to ask whether audio language models understand speech the way people encounter it: in long, messy, hour-long recordings rather than 30-second snippets. The paper's central claim is that they do not. Across eight tasks on 833+ hours of permissively licensed audio averaging 51 minutes, every evaluated model, including Gemini 2.0 Pro and GPT-4o, performs at or near floor level when asked to localize words, advertisements, or named entities, count speakers, estimate event or total durations, and rank emotional valence and arousal. The same models do much better on 30-second excerpts, so the paper attributes the failure primarily to audio length and to a reliance on the text prompt rather than on the acoustic signal.","feed_headline":"Hour-long audio stumps even top AI models","feed_subtitle":"A new 833-hour benchmark finds Gemini and GPT-4o near the floor on long-form audio reasoning.","key_machinery":"The load-bearing instrument is BLAB itself: 833+ hours of Creative Commons YouTube audio split into eight tasks across four categories, localization, counting, emotion, and duration, with roughly 200 question-audio pairs per task and an average audio length of 51 minutes. Annotation pipelines combine WhisperX forced alignment with human checking for word and advertisement localization, Claude-3-extracted entity spans mapped back to WhisperX timestamps for named-entity localization, and human-AI collaboration for emotion ranking. Evaluation uses word-level or frame-level F1 for localization and exact-match accuracy for counting, duration, and emotion, plus a 30-second BLAB-MINI subset that isolates the effect of audio length and noise-placement experiments that test whether models actually listen to the signal.","core_discovery":"On its own terms, the paper establishes that state-of-the-art audio language models have no reliable long-form audio understanding. On BLAB's localization tasks, the best reported word-level F1 is below 2%, and frame-level F1 for advertisement or named-entity localization stays below 5%; exact-match accuracy is below 9% for speaker number estimation, below 5% for event duration, and below 4% for whole-file duration. The models can sometimes detect that an event exists, with Gemini finding about 27% of ground-truth named entities, but they cannot place those events in time. On emotion reasoning, the best exact match is 64.29%, and valence ranking is near chance. The paper also shows that performance collapses with duration: truncating the same tasks to 30 seconds raises Gemini's word-localization F1 from about 1 to about 30 and its entity-localization F1 from roughly 2 to nearly 50, while replacing long audio with silence or Gaussian noise does not make models reliably answer zero.","pith_inferences":["The paper does not publish a no-audio prompt-only baseline, so its finding that models lean on prompts implies a testable extension: run BLAB questions with the audio replaced by silence and score residual accuracy, since any model that still answers above chance is effectively gambling on the prompt.","Because speaker-count ground truth is a range in 60% of cases, scoring models against individual annotators rather than the consensus range could reveal whether model errors concentrate where humans disagree, a sign of inherent ambiguity rather than model failure.","Gemini's degraded performance when a clean clip sits in the middle of noisy audio suggests an architectural probe: compare full-context listening with a chunk-then-aggregate pipeline on the same BLAB items to see whether the bottleneck is long-range attention or auditory perception.","If the benchmark's difficulty is confirmed, cascade systems built from transcription, forced alignment, and text search may beat end-to-end audio LMs on word and entity localization even though BLAB was designed to resist cascades; a direct head-to-head comparison would settle that."],"forward_implications":["A model that performs well on BLAB would, by construction, handle hour-scale audio localization, speaker counting, duration estimation, and emotion reasoning, so BLAB sets a concrete target for long-context audio training beyond short-clip benchmarks.","BLAB-MINI results imply that 30-second or 5-minute audio evaluations substantially overstate real-world capability, since Gemini's word-localization F1 drops from about 30 to about 1 when audio grows from 30 seconds to 30-plus minutes.","The noise-replacement and clip-placement analyses imply that long audio LMs can answer from the prompt alone and lose track of information in the middle of the context, so input ordering and prompt design matter as much as the acoustic signal.","Zero-shot chain-of-thought prompting raises event-duration accuracy only from about 4% to about 7% and speaker-count accuracy from about 8% to about 11%, so the identified bottleneck is auditory perception rather than high-level reasoning.","Most open-weight audio LMs cannot even ingest the long-form inputs BLAB requires, which means the open ecosystem is currently unevaluable on the very capability the benchmark targets."],"supporting_citations":[{"why":"Represents the short-audio benchmarks BLAB is contrasted against, with only about two entities per localization sample versus BLAB's average of 46.","marker":"Huang et al., 2024"},{"why":"Provides the closest prior benchmark, LongAudioBench, with a maximum audio length of five minutes, and supplies the Audio Flamingo 2 model evaluated on BLAB-MINI.","marker":"Ghosh et al., 2025"},{"why":"WhisperX supplies the word-level timestamps and transcripts from which word, named-entity, and advertisement localization labels are derived.","marker":"Bain et al., 2023"},{"why":"Claude-3 extracts the named-entity spans from WhisperX transcripts that become the ground truth for named-entity localization.","marker":"Anthropic, 2024"},{"why":"GPT-4 proposes advertisement segments for human review and is also one of the evaluated audio language models.","marker":"OpenAI et al., 2024"},{"why":"Defines the word-timing evaluation and the near-99% F1 reference point against which the paper measures BLAB's word-localization difficulty.","marker":"Sainath et al., 2020"},{"why":"Supplies the 'lost in the middle' result from text LMs that the paper's noise-placement analysis of Gemini on long audio echoes.","marker":"Liu et al., 2024"},{"why":"Provides the audio-based valence and arousal recognizer used to select emotion-ranking segments whose scores diverge from text-only estimates.","marker":"Wagner et al., 2023"}],"fun_headline_variants":["AI flubs hour-long audio placement tasks","Long audio: AI models lose the plot","Even top AI can't keep time on long audio","BLAB benchmark: AI's long audio blind spot","AI can't locate a sound in an hour of audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's named-entity localization ground truth is the weakest link because Claude-3, not a human, decided what counts as an entity, and if its extracted spans are incomplete or misaligned with WhisperX timestamps, models that actually detect those entities in the audio will be scored as wrong, making BLAB look harder than it is.","fun_headline_variants_meta":{"raw":{"variants":["AI flubs hour-long audio placement tasks","Long audio: AI models lose the plot","Even top AI can't keep time on long audio","BLAB benchmark: AI's long audio blind spot","AI can't locate a sound in an hour of audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001143,"raw_usage":{"total_tokens":4786,"prompt_tokens":1028,"completion_tokens":3758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":3685}},"tokens_in":644,"tokens_out":3758,"duration_ms":25073,"temperature":1.0,"reasoning_tokens":3685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:59:49.288632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of BLAB's named-entity localization items, have independent human annotators mark every entity span directly in the audio, and recompute frame-level F1 against both the human labels and the Claude-3/WhisperX labels; if models score substantially higher against human labels, or if the machine labels contain many spans humans cannot hear, then the reported NE-localization difficulty is partly an artifact of label generation rather than model limitation.","supporting_citations":[{"cited_title":"Dawn of the transformer era in speech emotion recognition: Closing the valence gap","cited_arxiv_id":null,"evidence_quote":"Provides the audio-based valence and arousal recognizer used to select emotion-ranking segments whose scores diverge from text-only estimates."}],"review_version":1}