Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning

Apoorva Kulkarni; Debarpan Bhattacharya; Sriram Ganapathy

REVIEW 5 major objections 4 minor 1 cited by

Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-source large audio language models consistently fall behind humans on temporal reasoning about sound events, and their accuracy scores do not predict how consistent they are under semantically identical perturbations.

desk verdict Useful, modest benchmark paper with a robust headline gap between LALMs and humans on audio temporal reasoning, but the uncertainty analysis and human baseline need tightening before the secondary claims should be trusted. read the letter →

arxiv 2505.13115 v1 pith:2V47UOUQ submitted 2025-05-19 cs.CL cs.AIcs.LGcs.SDeess.AS

Debarpan Bhattacharya , Apoorva Kulkarni , Sriram Ganapathy This is my paper · ORCID

classification cs.CLcs.AIcs.LGcs.SDeess.AS

keywords temporalreasoninglargeaudiolanguagemodelsbenchmarkeventorderingcountingdurationuncertaintyestimationmulti-choicequestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current open-source large audio language models (LALMs) cannot yet reason about the temporal structure of sound: which event came first, how many distinct sources are present, and how long each event lasts. To test this, the authors build TREA, a 600-question multiple-choice benchmark derived from ESC-50, and show every model they evaluate falls far behind human listeners (humans exceed 80% on all sub-tasks, while the best model stays below 50% on two of three). The paper also introduces an uncertainty score (EUE) measuring how often a model changes its answer when the input is perturbed in ways that keep the meaning the same. Across models, accuracy and this uncertainty score do not track each other, which suggests accuracy alone is a misleading summary of a model's reliability for high-stakes audio applications.

What carries the argument

The load-bearing object is TREA, a 600-sample multiple-choice question dataset (200 per sub-task) built by splicing ESC-50 recordings into multi-event clips, with sub-tasks for event duration, event ordering, and event counting. The second mechanism is the expected uncertainty estimate (EUE): for each test sample, 60 perturbations are produced by rephrasing the question and applying audio transformations (silence insertion, volume change, shuffling event order, altering duration, or inserting/deleting repetitions) that are supposed to preserve the ground-truth answer; the EUE is the average fraction of perturbations on which the model's answer changes. This measures local semantic consistency without needing ground-truth labels.

What would settle it

Take the TREA-C counting samples and for each perturbed version produced by aug insertion/deletion, have a human or a reliable oracle label how many unique sound sources are actually present; if a substantial fraction of perturbations have a different correct answer than the original clip, then the EUE for the counting task is not measuring semantic consistency, and the reported accuracy–uncertainty relationship for that sub-task would need revision.

Watch

Extended reading notes

Core claim

The central discovery is that open-source LALMs systematically fail audio temporal reasoning: on the TREA dataset, the best accuracy across all evaluated prompts is below 50% on the duration and counting sub-tasks, and only modestly above chance on ordering, while human listeners score 81–99%. A second finding is that measuring consistency under semantically grounded perturbations (EUE) reveals a different ranking of models than accuracy does: for instance, SALMONN-13B has lower calibration and uncertainty errors than the higher-accuracy SALMONN-plus-LLaMa pipeline. The authors conclude that temporal reasoning in audio is a distinct, unsolved capability, and that accuracy, calibration, and uncertainty must be reported together.

Load-bearing premise

The claim that accuracy and uncertainty are uncorrelated assumes that every perturbation leaves the ground-truth answer unchanged; for the counting sub-task, the insertion/deletion augmentation changes the number of repeated events, which can change the true number of unique sound sources and therefore the correct answer.

Editorial extensions

If this is right

If accuracy on TREA is representative, current LALMs cannot be trusted for applications that require answering when or how often sound events occur, such as surveillance review, meeting summaries, or media indexing.
Prompting strategies like chain-of-thought do not consistently improve audio temporal reasoning, so gains from text-based reasoning methods do not automatically transfer to the audio modality.
Using an LALM as a caption generator and then feeding captions to a text LLM can outperform direct audio question-answering, but the benefit depends heavily on which LALM produces the captions.
Reporting accuracy, calibration error, and uncertainty together gives a more complete picture of model reliability than accuracy alone, and different models can win on different metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The counting-task augmentation, which inserts or deletes repetitions of the same event class, may change the true number of unique sound sources and therefore the ground-truth answer to the counting question, so the EUE for TREA-C could be measuring answer sensitivity to a changed question rather than semantic uncertainty.
If EUE becomes a standard metric, model developers could explicitly train LALMs to be invariant to these semantic-preserving perturbations, potentially improving robustness without necessarily raising raw accuracy.
The TREA design of splicing short environmental clips could be extended to longer, more natural audio or to cross-modal temporal reasoning to test whether the observed gap persists outside controlled datasets.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper gives the field something it didn't have — a small, focused benchmark for temporal reasoning in audio — and the headline result, that open LALMs are far behind humans, is almost certainly right. The uncertainty analysis is a reasonable idea, but the evidence behind it is thinner, and one augmentation may invalidate the counting portion.

The TREA dataset is the real contribution. Six hundred MCQA items across three sub-tasks (duration, ordering, counting), built from ESC-50, with a comparison table showing that prior benchmarks like MMAU and AIR-Bench don't isolate these skills. That's a genuine gap, and Table 2 gives a useful map: ordering is the easiest for models, duration and counting are near-random for several models, while humans sit around 80-99%. Even with a noisy human baseline, a 30-50 point gap on ordering and counting isn't going to evaporate.

The soft spots are real but mostly fixable. The human baseline is six participants on 45 items with no stated sampling protocol or agreement stats; the paper should report that and ideally score the models on the same 45 items. The claim that the best model is below 50% on two tasks is sensitive to small-sample noise — 200 items gives wide binomial intervals — so it should be worded with uncertainty bounds. The EUE metric measures consistency with the model's own prediction, not correctness; the paper is explicit about that, but readers should not over-read the accuracy-uncertainty dissociation. The bigger problem is the counting augmentation: inserting or deleting repetitions of the same event class can change the answer to "how many unique sound sources," which breaks the label-preserving assumption for that sub-task. The authors need to fix that augmentation or drop counting from the EUE analysis. The lack of error bars anywhere and the 15-sample uncertainty subset are minor but should be addressed.

Citation pattern looks fine, and the related-work table is honest about what exists. The dataset/code link is under-specified with no commit hash, but the repository is named.

Who this is for: people benchmarking audio-language models and anyone building LALMs. It's not a methods paper, but the dataset alone is worth having. I'd send it to peer review — with the expectation that the counting augmentation and human-subset details get tightened, not a rewrite.

Referee Report

5 major / 4 minor

Summary. The paper introduces TREA, a new 600-sample audio multiple-choice dataset derived from ESC-50 that targets three fine-grained temporal reasoning subtasks: event duration (TREA-D), event ordering (TREA-O), and event counting (TREA-C). The authors benchmark three open-source large audio-language models (Qwen2-Audio, SALMONN, WavLLM) in four zero-shot settings (vanilla, chain-of-thought, explanation, and audio-description-plus-LLM-QA) and report that all models perform far below a human baseline obtained from six participants on a 45-question subset. The paper also proposes an expected uncertainty estimate (EUE) that measures a model's response invariance under semantically grounded test-time perturbations, reports expected calibration error (ECE), and concludes that accuracy, calibration, and uncertainty are not necessarily correlated. The dataset and code are publicly released.

Significance. The TREA dataset addresses a real gap in audio-language-model evaluation: existing audio benchmarks contain few temporal reasoning items, and a dedicated, publicly released benchmark with per-subtask granularity is a useful contribution. The test-time perturbation framework for EUE is also a valuable direction, and the public code and data support reproducibility. The large reported gap between LALMs and human listeners, if it holds, would be an important finding for the community. However, the human-baseline comparison and the uncertainty estimates are currently supported only weakly, so the paper's headline claims are conditional on additional statistical and methodological support.

major comments (5)

[Section 4, Table 2] The human baseline is not directly comparable to the model rows. The 92.2/98.9/81.1 human accuracies are computed on an unstated 45-question subset (15 per subtask) answered by 6 participants, while the LALM accuracies are computed on all 200 items per subtask. The paper does not describe how the 45 items were selected, whether participants were naive to the synthetic audio, or what the inter-participant agreement was; if the subset is easier than the full test set, the claim that LALMs are 'consistently behind human capabilities' is not strictly established. Please report the subset selection procedure, per-participant accuracy, and agreement statistics, or run the human evaluation on the full 200-item set.
[Section 4, Table 2] The statement that 'the best performance is < 50% on two out of the three TREA tasks' is not statistically robust at the reported sample sizes. With 200 items per subtask, the best duration accuracy of 47.0% and the best count accuracy of 45.5% have 95% binomial confidence intervals that include 50%, so below-chance or at-chance performance cannot be reliably distinguished from above-chance performance. Report binomial confidence intervals or a formal test for each accuracy value before asserting that the models are below 50%.
[Section 5.1.1] The aug_insertion_deletion() augmentation for the counting task is not label-preserving by construction. If the TREA-C question asks for the number of unique sound sources, randomizing the number of repetitions of the same event class may change the number of distinct source instances; if the question asks for the total number of events, it definitely changes the answer. The paper's blanket assertion that all perturbations leave the ground-truth label unchanged therefore needs explicit verification for this augmentation, and without it the counting-task EUE measures something other than semantic uncertainty.
[Section 5.1.2, Eq. (1)] EUE is defined relative to the model's own prediction on the original sample, not relative to the true label. A model that is confidently and consistently wrong will receive a low EUE, so the paper's conclusion that 'accuracy and uncertainty metrics are not necessarily correlated' is partly an artifact of this self-referential definition. The authors should report correctness-conditioned uncertainty (for example, EUE computed separately on samples where the model is correct versus incorrect) or compare EUE against perturbations whose label preservation is verified, and they should explicitly discuss this limitation.
[Section 5.1.1 and Table 3] The uncertainty evaluation uses only 15 original samples per subtask with 60 perturbations per sample, yet no error bars, confidence intervals, or sensitivity analyses are provided. With K=15, the EUE and ECE estimates in Table 3 have substantial sampling variability, and the combined values are not robust enough to support the cross-model comparison. Report bootstrap intervals or other uncertainty estimates, and clarify whether the 15 samples were randomly selected and how they relate to the human-evaluation subset.

minor comments (4)

[Section 5.1.2, Eq. (1)] The notation |i : M(x_i^n) != M(x_n)| inside the sum is ambiguous; it should be written as a indicator-based fraction, for example (1/N) * sum_{i=1}^N 1[M(x_i^n) != M(x_n)], to make the computation explicit.
[Section 2] The phrase 'epistamic uncertainty' contains a typo and should read 'epistemic uncertainty'.
[Table 3] The row label 'SALMONN+LLaMa-83B' appears truncated and should be 'SALMONN + LLaMa-3.3-70B' or another consistent label; the table should also clarify that the Acc. column is computed on perturbed samples, since this differs from the original-sample accuracy in Table 2.
[Section 5.1.1] The text states that textual modification involves rephrasing the question in multiple ways, but no examples, templates, or counts of the rephrasings are provided; including the actual paraphrase templates would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark, human comparison, and uncertainty metric are empirical definitions and external measurements, not derivations from their own outputs.

full rationale

The paper's central claim is an empirical benchmark result: LALMs are evaluated zero-shot on a new dataset (TREA) built from ESC-50 recordings, and their accuracy is compared with human accuracy on a subset of items. The dataset construction rules are explicit (duration differences, no repeated events for ordering, repeated events for counting) and do not encode the model outputs or the conclusions. No parameter is fitted to the benchmark data and then renamed as a prediction; the reported accuracies are direct measurements. The proposed uncertainty metric (EUE, Eq. 1) is defined as the fraction of semantically grounded perturbations for which the model's answer changes relative to its own answer on the original sample. This is a consistency statistic, not a claim that consistency implies correctness or calibration. Calling it 'self-referential' is a semantic observation about confidence measures in general, but it is not a circular derivation: the paper does not use EUE to predict accuracy, nor does it fit EUE to the accuracy numbers. The subsequent claim that accuracy and uncertainty are not correlated is an empirical observation from Table 3. There are no load-bearing self-citations; prior work is cited for frameworks (e.g., test-time augmentation, ECE) but the present results are generated from the paper's own dataset and evaluations. Potential concerns about the unmatched human subset size or perturbation preserving labels for counting are validity or correctness risks, not circularity. On the definitional criterion required here, no step reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central accuracy claim relies only on the dataset and zero-shot evaluation protocol. The uncertainty claim additionally relies on hand-chosen augmentation parameters, a small evaluation subset, and the assumption that all perturbations preserve the label. No new theoretical entities are introduced.

free parameters (3)

number of perturbations per sample = 60
Chosen by hand as 4 text rephrasings times 15 audio augmentations; the EUE metric depends on this count, but the central accuracy finding does not.
uncertainty evaluation subset size = 15 per task
Only 15 of 200 test samples per sub-task were used for EUE and ECE to limit compute; this small subset underpins the accuracy-uncertainty correlation claim.
augmentation hyperparameters = not specified
Random silence duration, volume scaling ranges, and rephrasing templates are described qualitatively, so the exact perturbation distribution is under-specified.

assumptions (4)

domain assumption Zero-shot prompting is a valid way to measure LALM capability
All model evaluations use zero-shot MCQA; results may reflect instruction-following limitations rather than reasoning ability, though CoT and explanation variants help control for this.
domain assumption Concatenated ESC-50 clips constitute valid temporal reasoning stimuli
The dataset is built by splicing 5-second environmental sound recordings with silence removal; this assumes the resulting audio contains natural, answerable temporal structure.
domain assumption Rephrasing questions and applying the listed audio transforms preserves semantic content and ground-truth label
The EUE metric relies on this; in particular, insertion/deletion of same-class repetitions may change the count of unique sources in the counting task.
domain assumption Token probabilities of the selected answer are meaningful confidence estimates
ECE is computed from the probability of the chosen MCQA option; this is standard but can be miscalibrated by decoding choices.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning." pith.science (2026). https://pith.science/paper/2V47UOUQ

@misc{pith2026250513115,
  author       = {Pith},
  title        = {Pith review of: Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2V47UOUQ}},
  note         = {Machine review of arXiv:2505.13115}
}

read the original abstract

The popular success of text-based large language models (LLM) has streamlined the attention of the multimodal community to combine other modalities like vision and audio along with text to achieve similar multimodal capabilities. In this quest, large audio language models (LALMs) have to be evaluated on reasoning related tasks which are different from traditional classification or generation tasks. Towards this goal, we propose a novel dataset called temporal reasoning evaluation of audio (TREA). We benchmark open-source LALMs and observe that they are consistently behind human capabilities on the tasks in the TREA dataset. While evaluating LALMs, we also propose an uncertainty metric, which computes the invariance of the model to semantically identical perturbations of the input. Our analysis shows that the accuracy and uncertainty metrics are not necessarily correlated and thus, points to a need for wholesome evaluation of LALMs for high-stakes applications.

Figures

Figures reproduced from arXiv: 2505.13115 by the authors.

**Figure 1.** Examples from TREA dataset illustrating the subtasks in event order, count and duration based MCQA. probed which involved speech, music and sound classification tasks. In spite of these efforts, temporal reasoning capabilities are largely unexplored. However, in the image domain, visual spatial reasoning has been probed elaborately through works like [29] and with novel datasets like BLINK [16]. In this paper, we … view at source ↗

**Figure 2.** Examples of original sample (from order task), and two perturbed samples, one with audio modification and one with text modification. the meaning is unchanged (aug paraphrase()). For audio perturbations, we use various augmentation strategies. • aug silence(): This operation inserts silence (of random duration) at various positions (randomly chosen) between sound events. This augmentation is used in all tasks. • a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning
cs.SD 2026-07 conditional novelty 6.0 of 10

A self-play game with a known 'odd listener' converts unlabeled audio contrast pairs into a verifiable reward, improving fine-grained audio reasoning on TREA, MMAU, and MMAR.

Reference graph

Works this paper leans on

48 extracted references · 23 canonical work pages · cited by 1 Pith paper

[1]

Introduction The development of text-based large language models (LLM) like GPT [1], LLAMA [2], Gemini [3], Mistral [4], etc., have pushed the envelope on text processing capabilities, driven by unsupervised pre-training on humongous amount of text corpus crawled from the internet. These models have made tremendous advances in achieving human and super hu...

work page 2025
[2]

Related Work and Contributions Large Audio Language Models (LALMs): The early work on extending text-based LLMs to audio domain can be traced to ef- forts like Audio-PALM [22], CLAP [23] and Speech-GPT [24]. With the popularization of the open-source LLaMa based mod- els [2] and with the availability of open-source speech recog- nition systems (Whisper - ...

work page Pith review arXiv 2025
[3]

can opening

Dataset Design We design a dataset to benchmark temporal reasoning tasks in audio, termed temporal reasoning evaluation of audio (TREA). It probes the models on three sub-tasks - event duration (TREA- D), ordering (TREA-O) and counting (TREA-C). The dataset size is 600 samples, with each sub-task containing200 samples. The dataset is derived from the Envi...

work page
[4]

The LALMs are evaluated in zero-shot manner and the performance (accu- racy (%)) is reported in Table 2

Benchmarking and analysis We consider audio temporal reasoning subset of MMAU dataset [20] (48 samples), and the proposed TREA dataset (total 600 samples spread across 3 fine grained tasks). The LALMs are evaluated in zero-shot manner and the performance (accu- racy (%)) is reported in Table 2. The top row (random) denotes an experiment where one of the4 ...

work page
[5]

Test-time Uncertainty Measure We propose to measure the uncertainty in the decision making for a test sample using data perturbations

Metrics Beyond Accuracy 5.1. Test-time Uncertainty Measure We propose to measure the uncertainty in the decision making for a test sample using data perturbations. The perturbations are generated in such a way that the semantic content of the perturbed sample is identical to the original test sample (unal- tered ground truth label) The analogy in image cl...

work page
[6]

Summary In this paper, we have proposed multiple novel components that help advance the understanding and benchmarking of LALMs. First, the work proposes a novel dataset named temporal rea- soning evaluation of audio (TREA) which allows fine-grained assessment of the LALM’s capabilities on order, duration and counting of audio events separately. Second, t...

work page
[7]

MM-LLMs: Recent advances in multi- modal large language models,

D. Zhang, Y . Yu et al., “MM-LLMs: Recent advances in multi- modal large language models,” arXiv preprint arXiv:2401.13601, 2024

arXiv 2024
[8]

GPT-4 technical report,

J. Achiam, S. Adler et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

Show all 48 references

[9]

LLaMa: Open and efficient founda- tion language models,

H. Touvron, T. Lavril et al., “LLaMa: Open and efficient founda- tion language models,” arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[10]

Gemini: a family of highly capable mul- timodal models,

G. Team, R. Anil et al., “Gemini: a family of highly capable mul- timodal models,” arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[11]

Mistral 7b,

A. Q. Jiang, A. Sablayrolles et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023

2023 arXiv
[12]

Measuring massive multitask lan- guage understanding,

D. Hendrycks, C. Burns et al., “Measuring massive multitask lan- guage understanding,” in International Conference on Learning Representations, 2024

2024
[13]

Levels of AGI: Op- erationalizing progress on the path to agi,

M. R. Morris, J. Sohl-Dickstein et al. , “Levels of AGI: Op- erationalizing progress on the path to agi,” arXiv preprint arXiv:2311.02462, 2023

2023
[14]

A survey on evaluation of large lan- guage models,

Y . Chang, X. Wang et al., “A survey on evaluation of large lan- guage models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024

2024
[15]

Visual instruction tuning,

H. Liu, C. Li et al., “Visual instruction tuning,” Advances in neu- ral information processing systems, vol. 36, 2024

2024
[16]

Llava++: Extending visual capabilities with llama-3 and phi-3,

H. Rasheed, M. Maaz et al. , “Llava++: Extending visual capabilities with llama-3 and phi-3,” 2024. [Online]. Available: https://github.com/mbzuai-oryx/LLaV A-pp

2024
[17]

Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,

Y . Chu, J. Xu et al., “Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

2023 arXiv
[18]

SALMONN: Towards generic hearing abilities for large language models,

C. Tang, W. Yu et al. , “SALMONN: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

2023 arXiv
[19]

WavLLM: Towards robust and adaptive speech large language model,

S. Hu, L. Zhou et al., “WavLLM: Towards robust and adaptive speech large language model,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , Nov. 2024, pp. 4552– 4572

2024
[20]

Chatbot arena: An open plat- form for evaluating llms by human preference,

W.-L. Chiang, L. Zheng et al. , “Chatbot arena: An open plat- form for evaluating llms by human preference,” arXiv preprint arXiv:2403.04132, 2024

2024 arXiv
[21]

What do speech foundation models not learn about speech?

A. Waheed, H. Atwany et al., “What do speech foundation models not learn about speech?” arXiv preprint arXiv:2410.12948, 2024

2024 arXiv
[22]

LVLM: A comprehensive evaluation benchmark for large vision-language models,

P. Xu, W. Shao et al. , “LVLM: A comprehensive evaluation benchmark for large vision-language models,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

2024
[23]

BLINK: Multimodal large language models can see but not perceive,

X. Fu, Y . Hu et al., “BLINK: Multimodal large language models can see but not perceive,” in European Conference on Computer Vision. Springer, 2024, pp. 148–166

2024
[24]

Ttslow: Slow down text-to-speech with efficiency robustness evaluations,

X. Gao, Y . Chen et al., “Ttslow: Slow down text-to-speech with efficiency robustness evaluations,” IEEE Transactions on Audio, Speech and Language Processing, 2025

2025
[25]

Can Whisper perform speech- based in-context learning?

S. Wang, C.-H. Yang et al. , “Can Whisper perform speech- based in-context learning?” in ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 13 421–13 425

2024
[26]

Recent advances in speech language models: A survey,

W. Cui, D. Yu et al., “Recent advances in speech language models: A survey,”arXiv preprint arXiv:2410.03751, 2024

2024 arXiv
[27]

MMAU: A massive multi-task au- dio understanding and reasoning benchmark,

S. Sakshi, U. Tyagi et al. , “MMAU: A massive multi-task au- dio understanding and reasoning benchmark,” arXiv preprint arXiv:2410.19168, 2024

2024 arXiv
[28]

AIR-Bench: Benchmarking large audio- language models via generative comprehension,

Q. Yang, J. Xu et al., “AIR-Bench: Benchmarking large audio- language models via generative comprehension,” arXiv preprint arXiv:2402.07729, 2024

2024 arXiv
[29]

AudioPALM: A large language model that can speak and listen,

P. K. Rubenstein, C. Asawaroengchai et al. , “AudioPALM: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925, 2023

2023 arXiv
[30]

CLAP learning audio concepts from natural language supervision,

B. Elizalde, S. Deshmukh et al., “CLAP learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2023, pp. 1–5

2023
[31]

SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,

D. Zhang, S. Li et al., “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,” arXiv preprint arXiv:2305.11000, 2023

2023 arXiv
[32]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim et al. , “Robust speech recognition via large-scale weak supervision,” inInternational conference on ma- chine learning. PMLR, 2023, pp. 28 492–28 518

2023
[33]

In deep learning based image classification setting, a similar uncertainty measure was proposed [34]

which measures the resilience of the model to semanti- cally invariant perturbations of the input. In deep learning based image classification setting, a similar uncertainty measure was proposed [34]. In this paper, we adapt this framework for audio- based uncertainty estimati...
[34]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen et al. , “LoRA: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[35]

Beyond single-audio: Advancing multi- audio processing in audio large language models,

Y . Chen, X. Yue et al., “Beyond single-audio: Advancing multi- audio processing in audio large language models,” arXiv preprint arXiv:2409.18680, 2024

2024 arXiv
[36]

Is a picture worth a thousand words? delving into spatial reasoning for vision language models,

J. Wang, Y . Ming et al., “Is a picture worth a thousand words? delving into spatial reasoning for vision language models,” Ad- vances in Neural Information Processing Systems , vol. 37, pp. 75 392–75 421, 2025

2025
[37]

A survey of confidence estimation and cal- ibration in large language models,

J. Geng, F. Cai et al., “A survey of confidence estimation and cal- ibration in large language models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers), ...

2024
[38]

Scaling up visual and vision-language rep- resentation learning with noisy text supervision,

C. Jia, Y . Yang et al., “Scaling up visual and vision-language rep- resentation learning with noisy text supervision,” in International conference on machine learning. PMLR, 2021, pp. 4904–4916

2021
[39]

Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation,

L. Kuhn, Y . Gal et al., “Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation,” arXiv preprint arXiv:2302.09664, 2023

2023 arXiv
[40]

A survey on uncertainty quantifica- tion of large language models: Taxonomy, open research chal- lenges, and future directions,

O. Shorinwa, Z. Mei et al., “A survey on uncertainty quantifica- tion of large language models: Taxonomy, open research chal- lenges, and future directions,” arXiv preprint arXiv:2412.05563, 2024

2024 arXiv
[41]

Classification confidence estimation with test-time data-augmentation,

Y . Bahat and G. Shakhnarovich, “Classification confidence estimation with test-time data-augmentation,” arXiv preprint arXiv:2006.16705, 2020

2006 arXiv
[42]

Gama: A large audio-language model with advanced audio understanding and complex reasoning abili- ties,

S. Ghosh, S. Kumar et al., “Gama: A large audio-language model with advanced audio understanding and complex reasoning abili- ties,” arXiv preprint arXiv:2406.11768, 2024

2024 arXiv
[43]

Mustango: Toward control- lable text-to-music generation,

J. Melechovsky, Z. Guo et al. , “Mustango: Toward control- lable text-to-music generation,”arXiv preprint arXiv:2311.08355, 2023

2023 arXiv
[44]

Muchomusic: Evaluating music under- standing in multimodal audio-language models,

B. Weck, I. Manco et al., “Muchomusic: Evaluating music under- standing in multimodal audio-language models,” arXiv preprint arXiv:2408.01337, 2024

2024 arXiv
[45]

Audiobench: A universal benchmark for audio large language models,

B. Wang, X. Zou et al., “Audiobench: A universal benchmark for audio large language models,” arXiv preprint arXiv:2406.16020, 2024

2024 arXiv
[46]

ESC: Dataset for environmental sound classifica- tion,

K. J. Piczak, “ESC: Dataset for environmental sound classifica- tion,” in Proceedings of the 23rd ACM international conference on Multimedia, 2015, pp. 1015–1018

2015
[47]

Chain-of-thought prompting elicits rea- soning in large language models,

J. Wei, X. Wang et al., “Chain-of-thought prompting elicits rea- soning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

2022
[48]

How can we know when language mod- els know? on the calibration of language models for question an- swering,

Z. Jiang, J. Araki et al., “How can we know when language mod- els know? on the calibration of language models for question an- swering,” Transactions of the Association for Computational Lin- guistics, vol. 9, pp. 962–977, 2021

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Introduction The development of text-based large language models (LLM) like GPT [1], LLAMA [2], Gemini [3], Mistral [4], etc., have pushed the envelope on text processing capabilities, driven by unsupervised pre-training on humongous amount of text corpus crawled from the internet. These models have made tremendous advances in achieving human and super hu...

work page 2025

[2] [2]

Related Work and Contributions Large Audio Language Models (LALMs): The early work on extending text-based LLMs to audio domain can be traced to ef- forts like Audio-PALM [22], CLAP [23] and Speech-GPT [24]. With the popularization of the open-source LLaMa based mod- els [2] and with the availability of open-source speech recog- nition systems (Whisper - ...

work page Pith review arXiv 2025

[3] [3]

can opening

Dataset Design We design a dataset to benchmark temporal reasoning tasks in audio, termed temporal reasoning evaluation of audio (TREA). It probes the models on three sub-tasks - event duration (TREA- D), ordering (TREA-O) and counting (TREA-C). The dataset size is 600 samples, with each sub-task containing200 samples. The dataset is derived from the Envi...

work page

[4] [4]

The LALMs are evaluated in zero-shot manner and the performance (accu- racy (%)) is reported in Table 2

Benchmarking and analysis We consider audio temporal reasoning subset of MMAU dataset [20] (48 samples), and the proposed TREA dataset (total 600 samples spread across 3 fine grained tasks). The LALMs are evaluated in zero-shot manner and the performance (accu- racy (%)) is reported in Table 2. The top row (random) denotes an experiment where one of the4 ...

work page

[5] [5]

Test-time Uncertainty Measure We propose to measure the uncertainty in the decision making for a test sample using data perturbations

Metrics Beyond Accuracy 5.1. Test-time Uncertainty Measure We propose to measure the uncertainty in the decision making for a test sample using data perturbations. The perturbations are generated in such a way that the semantic content of the perturbed sample is identical to the original test sample (unal- tered ground truth label) The analogy in image cl...

work page

[6] [6]

Summary In this paper, we have proposed multiple novel components that help advance the understanding and benchmarking of LALMs. First, the work proposes a novel dataset named temporal rea- soning evaluation of audio (TREA) which allows fine-grained assessment of the LALM’s capabilities on order, duration and counting of audio events separately. Second, t...

work page

[7] [7]

MM-LLMs: Recent advances in multi- modal large language models,

D. Zhang, Y . Yu et al., “MM-LLMs: Recent advances in multi- modal large language models,” arXiv preprint arXiv:2401.13601, 2024

arXiv 2024

[8] [8]

GPT-4 technical report,

J. Achiam, S. Adler et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

arXiv 2023