{"id":"f4d07d9d-512a-45d5-9a22-d7c27e549adc","arxiv_id":"2505.13115","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Open-source audio-language models perform far below humans on a new 600-item benchmark of temporal reasoning in sound, and their accuracy does not track a proposed perturbation-based uncertainty measure.","lead":"This paper introduces TREA, a set of 600 audio multiple-choice questions that test whether AI models can reason about how long, in what order, and how many times sounds occur. It finds that open-source audio-language models score far below humans on these tasks, and proposes a perturbation-based uncertainty metric to measure how stable model answers are.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central human-vs-LALM comparison uses a 45-item human subset with unstated selection and no error bars, so the headline gap is not strictly established as reported.","rationale":"The reader's weakest assumption focused on the counting-task augmentation invalidating labels in the uncertainty metric; that is a valid secondary-claim concern. My strongest concern targets the primary human-comparison claim directly: the reported human baseline is measured on a different, much smaller, and uncharacterized subset from the model evaluation. This concern is partially reflected in the reader's rationale (human baseline from only 6 participants on 45 questions), so agreement is partial rather than full. The reader's CONDITIONAL verdict already captures the need for more rigor, and my stress-test does not move that verdict; it sharpens the reason why the main human-gap claim needs a matched-item check.","tokens_in":8294,"tokens_out":10019,"duration_ms":105530,"concrete_test":"Have the authors release the exact 45 human-answered items and have at least 10 naive participants answer a stratified random sample of 60+ items per TREA sub-task from the full dataset. Score the best LALM pipeline (e.g., SALMONN+LLaMa) on those same items and report binomial 95% confidence intervals for human and model accuracy. If the lower human confidence bound remains at least 20 points above the model point estimate on duration and counting, the headline gap is confirmed; if the interval overlaps, the comparison needs to be re-run on matched items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 compares human accuracy computed on a 45-question subset (15 per sub-task, 6 participants) to LALM accuracy computed on all 200 items per sub-task. The paper does not state how the human subset was sampled, whether the participants were naive, or what the inter-participant agreement was. If the 45 items were selected for clarity, the human numbers (92.2/98.9/81.1) are not directly comparable to model scores on the full task, and the claim that LALMs are 'consistently behind human capabilities' is not strictly supported by the evidence as presented. The gap is large, so this is unlikely to reverse the qualitative conclusion, but it is the least-secure load-bearing link in the strongest claim. A second numerical issue: with 200 items per task, the reported best duration accuracy 47.0% and count accuracy 45.5% have 95% binomial CIs that include 50% (roughly 40-54 and 39-52), so the specific 'best performance <50% on two of three tasks' wording is not statistically robust even though the distance from the human scores is clear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TREA, a new 600-sample audio multiple-choice dataset derived from ESC-50 that targets three fine-grained temporal reasoning subtasks: event duration (TREA-D), event ordering (TREA-O), and event counting (TREA-C). The authors benchmark three open-source large audio-language models (Qwen2-Audio, SALMONN, WavLLM) in four zero-shot settings (vanilla, chain-of-thought, explanation, and audio-description-plus-LLM-QA) and report that all models perform far below a human baseline obtained from six participants on a 45-question subset. The paper also proposes an expected uncertainty estimate (EUE) that measures a model's response invariance under semantically grounded test-time perturbations, reports expected calibration error (ECE), and concludes that accuracy, calibration, and uncertainty are not necessarily correlated. The dataset and code are publicly released.","tokens_in":8437,"tokens_out":4996,"duration_ms":49711,"significance":"The TREA dataset addresses a real gap in audio-language-model evaluation: existing audio benchmarks contain few temporal reasoning items, and a dedicated, publicly released benchmark with per-subtask granularity is a useful contribution. The test-time perturbation framework for EUE is also a valuable direction, and the public code and data support reproducibility. The large reported gap between LALMs and human listeners, if it holds, would be an important finding for the community. However, the human-baseline comparison and the uncertainty estimates are currently supported only weakly, so the paper's headline claims are conditional on additional statistical and methodological support.","major_comments":[{"comment":"The human baseline is not directly comparable to the model rows. The 92.2/98.9/81.1 human accuracies are computed on an unstated 45-question subset (15 per subtask) answered by 6 participants, while the LALM accuracies are computed on all 200 items per subtask. The paper does not describe how the 45 items were selected, whether participants were naive to the synthetic audio, or what the inter-participant agreement was; if the subset is easier than the full test set, the claim that LALMs are 'consistently behind human capabilities' is not strictly established. Please report the subset selection procedure, per-participant accuracy, and agreement statistics, or run the human evaluation on the full 200-item set.","section":"Section 4, Table 2"},{"comment":"The statement that 'the best performance is < 50% on two out of the three TREA tasks' is not statistically robust at the reported sample sizes. With 200 items per subtask, the best duration accuracy of 47.0% and the best count accuracy of 45.5% have 95% binomial confidence intervals that include 50%, so below-chance or at-chance performance cannot be reliably distinguished from above-chance performance. Report binomial confidence intervals or a formal test for each accuracy value before asserting that the models are below 50%.","section":"Section 4, Table 2"},{"comment":"The aug_insertion_deletion() augmentation for the counting task is not label-preserving by construction. If the TREA-C question asks for the number of unique sound sources, randomizing the number of repetitions of the same event class may change the number of distinct source instances; if the question asks for the total number of events, it definitely changes the answer. The paper's blanket assertion that all perturbations leave the ground-truth label unchanged therefore needs explicit verification for this augmentation, and without it the counting-task EUE measures something other than semantic uncertainty.","section":"Section 5.1.1"},{"comment":"EUE is defined relative to the model's own prediction on the original sample, not relative to the true label. A model that is confidently and consistently wrong will receive a low EUE, so the paper's conclusion that 'accuracy and uncertainty metrics are not necessarily correlated' is partly an artifact of this self-referential definition. The authors should report correctness-conditioned uncertainty (for example, EUE computed separately on samples where the model is correct versus incorrect) or compare EUE against perturbations whose label preservation is verified, and they should explicitly discuss this limitation.","section":"Section 5.1.2, Eq. (1)"},{"comment":"The uncertainty evaluation uses only 15 original samples per subtask with 60 perturbations per sample, yet no error bars, confidence intervals, or sensitivity analyses are provided. With K=15, the EUE and ECE estimates in Table 3 have substantial sampling variability, and the combined values are not robust enough to support the cross-model comparison. Report bootstrap intervals or other uncertainty estimates, and clarify whether the 15 samples were randomly selected and how they relate to the human-evaluation subset.","section":"Section 5.1.1 and Table 3"}],"minor_comments":[{"comment":"The notation |i : M(x_i^n) != M(x_n)| inside the sum is ambiguous; it should be written as a indicator-based fraction, for example (1/N) * sum_{i=1}^N 1[M(x_i^n) != M(x_n)], to make the computation explicit.","section":"Section 5.1.2, Eq. (1)"},{"comment":"The phrase 'epistamic uncertainty' contains a typo and should read 'epistemic uncertainty'.","section":"Section 2"},{"comment":"The row label 'SALMONN+LLaMa-83B' appears truncated and should be 'SALMONN + LLaMa-3.3-70B' or another consistent label; the table should also clarify that the Acc. column is computed on perturbed samples, since this differs from the original-sample accuracy in Table 2.","section":"Table 3"},{"comment":"The text states that textual modification involves rephrasing the question in multiple ways, but no examples, templates, or counts of the rephrasings are provided; including the actual paraphrase templates would improve reproducibility.","section":"Section 5.1.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives the field something it didn't have — a small, focused benchmark for temporal reasoning in audio — and the headline result, that open LALMs are far behind humans, is almost certainly right. The uncertainty analysis is a reasonable idea, but the evidence behind it is thinner, and one augmentation may invalidate the counting portion.\n\nThe TREA dataset is the real contribution. Six hundred MCQA items across three sub-tasks (duration, ordering, counting), built from ESC-50, with a comparison table showing that prior benchmarks like MMAU and AIR-Bench don't isolate these skills. That's a genuine gap, and Table 2 gives a useful map: ordering is the easiest for models, duration and counting are near-random for several models, while humans sit around 80-99%. Even with a noisy human baseline, a 30-50 point gap on ordering and counting isn't going to evaporate.\n\nThe soft spots are real but mostly fixable. The human baseline is six participants on 45 items with no stated sampling protocol or agreement stats; the paper should report that and ideally score the models on the same 45 items. The claim that the best model is below 50% on two tasks is sensitive to small-sample noise — 200 items gives wide binomial intervals — so it should be worded with uncertainty bounds. The EUE metric measures consistency with the model's own prediction, not correctness; the paper is explicit about that, but readers should not over-read the accuracy-uncertainty dissociation. The bigger problem is the counting augmentation: inserting or deleting repetitions of the same event class can change the answer to \"how many unique sound sources,\" which breaks the label-preserving assumption for that sub-task. The authors need to fix that augmentation or drop counting from the EUE analysis. The lack of error bars anywhere and the 15-sample uncertainty subset are minor but should be addressed.\n\nCitation pattern looks fine, and the related-work table is honest about what exists. The dataset/code link is under-specified with no commit hash, but the repository is named.\n\nWho this is for: people benchmarking audio-language models and anyone building LALMs. It's not a methods paper, but the dataset alone is worth having. I'd send it to peer review — with the expectation that the counting augmentation and human-subset details get tightened, not a rewrite.","headline":"Useful, modest benchmark paper with a robust headline gap between LALMs and humans on audio temporal reasoning, but the uncertainty analysis and human baseline need tightening before the secondary claims should be trusted.","tokens_in":9057,"tokens_out":3677,"would_cite":true,"duration_ms":40494,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source large audio language models consistently fall behind humans on temporal reasoning about sound events, and their accuracy scores do not predict how consistent they are under semantically identical perturbations.","keywords":["temporal reasoning","large audio language models","audio benchmark","event ordering","event counting","event duration","uncertainty estimation","multi-choice question answering"],"falsifier":"Take the TREA-C counting samples and for each perturbed version produced by aug insertion/deletion, have a human or a reliable oracle label how many unique sound sources are actually present; if a substantial fraction of perturbations have a different correct answer than the original clip, then the EUE for the counting task is not measuring semantic consistency, and the reported accuracy–uncertainty relationship for that sub-task would need revision.","tokens_in":8017,"feed_emoji":"🎧","tokens_out":3899,"duration_ms":36334,"temperature":0.7,"pith_summary":"This paper argues that current open-source large audio language models (LALMs) cannot yet reason about the temporal structure of sound: which event came first, how many distinct sources are present, and how long each event lasts. To test this, the authors build TREA, a 600-question multiple-choice benchmark derived from ESC-50, and show every model they evaluate falls far behind human listeners (humans exceed 80% on all sub-tasks, while the best model stays below 50% on two of three). The paper also introduces an uncertainty score (EUE) measuring how often a model changes its answer when the input is perturbed in ways that keep the meaning the same. Across models, accuracy and this uncertainty score do not track each other, which suggests accuracy alone is a misleading summary of a model's reliability for high-stakes audio applications.","feed_headline":"Best open-source audio AI scores below 50% on temporal reasoning","feed_subtitle":"New TREA benchmark: ordering, counting, and duration questions stump models; accuracy alone hides their uncertainty.","key_machinery":"The load-bearing object is TREA, a 600-sample multiple-choice question dataset (200 per sub-task) built by splicing ESC-50 recordings into multi-event clips, with sub-tasks for event duration, event ordering, and event counting. The second mechanism is the expected uncertainty estimate (EUE): for each test sample, 60 perturbations are produced by rephrasing the question and applying audio transformations (silence insertion, volume change, shuffling event order, altering duration, or inserting/deleting repetitions) that are supposed to preserve the ground-truth answer; the EUE is the average fraction of perturbations on which the model's answer changes. This measures local semantic consistency without needing ground-truth labels.","core_discovery":"The central discovery is that open-source LALMs systematically fail audio temporal reasoning: on the TREA dataset, the best accuracy across all evaluated prompts is below 50% on the duration and counting sub-tasks, and only modestly above chance on ordering, while human listeners score 81–99%. A second finding is that measuring consistency under semantically grounded perturbations (EUE) reveals a different ranking of models than accuracy does: for instance, SALMONN-13B has lower calibration and uncertainty errors than the higher-accuracy SALMONN-plus-LLaMa pipeline. The authors conclude that temporal reasoning in audio is a distinct, unsolved capability, and that accuracy, calibration, and uncertainty must be reported together.","pith_inferences":["The counting-task augmentation, which inserts or deletes repetitions of the same event class, may change the true number of unique sound sources and therefore the ground-truth answer to the counting question, so the EUE for TREA-C could be measuring answer sensitivity to a changed question rather than semantic uncertainty.","If EUE becomes a standard metric, model developers could explicitly train LALMs to be invariant to these semantic-preserving perturbations, potentially improving robustness without necessarily raising raw accuracy.","The TREA design of splicing short environmental clips could be extended to longer, more natural audio or to cross-modal temporal reasoning to test whether the observed gap persists outside controlled datasets."],"forward_implications":["If accuracy on TREA is representative, current LALMs cannot be trusted for applications that require answering when or how often sound events occur, such as surveillance review, meeting summaries, or media indexing.","Prompting strategies like chain-of-thought do not consistently improve audio temporal reasoning, so gains from text-based reasoning methods do not automatically transfer to the audio modality.","Using an LALM as a caption generator and then feeding captions to a text LLM can outperform direct audio question-answering, but the benefit depends heavily on which LALM produces the captions.","Reporting accuracy, calibration error, and uncertainty together gives a more complete picture of model reliability than accuracy alone, and different models can win on different metrics."],"supporting_citations":[{"why":"Supplies the ESC-50 environmental sound recordings from which TREA clips are assembled.","marker":"[39]"},{"why":"Provides the MMAU baseline whose temporal reasoning subset (48 samples) is compared against TREA.","marker":"[20]"},{"why":"One of the three LALMs benchmarked on TREA and MMAU.","marker":"[10]"},{"why":"One of the benchmarked LALMs; also used as caption generator in the description+LLM-QA setting.","marker":"[11]"},{"why":"One of the benchmarked LALMs.","marker":"[12]"},{"why":"Source of the test-time data augmentation approach adapted for the EUE uncertainty metric.","marker":"[34]"},{"why":"LLaMa backbone model used for the description+LLM-QA pipeline.","marker":"[2]"},{"why":"Defines the expected calibration error (ECE) used as the calibration metric.","marker":"[41]"}],"fun_headline_variants":["Open-source audio AI botches temporal reasoning on new TREA test","Audio LLMs underperform humans on temporal reasoning benchmark","TREA: Audio models fail timing tasks; uncertainty metric shifts rankings","Audio models lag humans on TREA temporal reasoning; uncertainty matters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that accuracy and uncertainty are uncorrelated assumes that every perturbation leaves the ground-truth answer unchanged; for the counting sub-task, the insertion/deletion augmentation changes the number of repeated events, which can change the true number of unique sound sources and therefore the correct answer.","fun_headline_variants_meta":{"raw":{"variants":["Open-source audio AI botches temporal reasoning on new TREA test","Audio LLMs underperform humans on temporal reasoning benchmark","TREA: Audio models fail timing tasks; uncertainty metric shifts rankings","Audio models lag humans on TREA temporal reasoning; uncertainty matters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00041,"raw_usage":{"total_tokens":2070,"prompt_tokens":832,"completion_tokens":1238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1166}},"tokens_in":448,"tokens_out":1238,"duration_ms":9198,"temperature":1.0,"reasoning_tokens":1166,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:19:23.761808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the TREA-C counting samples and for each perturbed version produced by aug insertion/deletion, have a human or a reliable oracle label how many unique sound sources are actually present; if a substantial fraction of perturbations have a different correct answer than the original clip, then the EUE for the counting task is not measuring semantic consistency, and the reported accuracy–uncertainty relationship for that sub-task would need revision.","supporting_citations":[{"cited_title":"Measuring massive multitask lan- guage understanding,","cited_arxiv_id":null,"evidence_quote":"One of the benchmarked LALMs."},{"cited_title":"Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning","cited_arxiv_id":"2505.13115","evidence_quote":"LLaMa backbone model used for the description+LLM-QA pipeline."}],"review_version":1}