Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents ADU-Bench, a 20,715-dialogue benchmark for open-ended audio dialogue understanding, and uses it to show that current large audio-language models systematically fail at math-heavy content, roleplay and common sense…

desk verdict A genuinely useful audio dialogue benchmark with a novel ambiguity axis, but the central prosody result rests on synthetic audio that is only validated on the general split, not the ambiguity items. read the letter →

arxiv 2412.05167 v2 pith:6BBDULYJ submitted 2024-12-06 cs.AI cs.CLcs.SDeess.AS

Kuofeng Gao , Shu-Tao Xia , Ke Xu , Philip Torr , Jindong Gu This is my paper · ORCID
classification cs.AIcs.CLcs.SDeess.AS
keywords audiodialogueunderstandinglargeaudio-languagemodelsADU-BenchspokenambiguityintonationandpausemultilingualLLM-as-judgesyntheticspeechbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to give the field of large audio-language models (LALMs) a measuring instrument for what it is that users actually do: hold open-ended spoken conversations. The authors construct ADU-Bench, four datasets totaling 20,715 audio dialogues, spanning 3 general scenarios, 12 skill domains, 9 languages, and 4 types of phonetic ambiguity. Running 16 LALMs through it, they find the same sentence said with a different intonation or a different pause placement is systematically misread, mathematics and code suffer when spoken aloud, non-English languages drop sharply, and even the strongest model reaches only moderate scores on ambiguity. The paper's contribution is the benchmark and the pattern of failures it exposes, which together give developers concrete targets for the next generation of audio dialogue models.

What carries the argument

The central object is ADU-Bench, a collection of (audio query, textual reference) tuples organized into four datasets: ADU-General (12,000 dialogues), ADU-Skill (3,725), ADU-Multilingual (3,600), and ADU-Ambiguity (1,390). The mechanism that carries the argument is a two-stage evaluation pipeline: first, LALMs receive the audio and produce a response; then a GPT-4 evaluator scores that response against a GPT-4-generated reference on a 0–10 scale, using the transcriptions as the query. To counter position bias, the pipeline runs the scoring twice with reference and response swapped and averages the two scores, and it cross-validates the judge against LLaMA-3-70B, Qwen-2-72B, and human pairwise comparisons. The ambiguity dataset is built on SSML speech synthesis, with prosody tags for intonation and break tags for pauses, which is what lets the same written sentence become audibly different utterances.

What would settle it

Have human listeners mark the intended meaning of ADU-Ambiguity audio without seeing the reference; if agreement is low, or if re-recording the same items with human voice actors reverses model rankings, the claim that LALMs specifically fail on phonetic ambiguity would not be supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, ADU-Bench establishes that open-ended audio dialogue understanding is a measurable capability distinct from speech recognition and from text instruction-following, and that today's LALMs are far better at fluent conversation than at catching the meaning carried by sound itself. The headline result is that the same literal sentence can express different intentions depending on intonation, pause position, or homophones, and that models—GPT-4o included—miss these cues: GPT-4o averages 8.16 overall but only 6.87 on the ambiguity dataset, while open-source models linger below 4 on a 0–10 scale. The skill results show mathematics, physics, chemistry, and coding score lowest, which the authors attribute to mathematical symbols and formulas being hard to follow in audio; common sense and roleplay are also weak, pointing to missing understanding of human behavior. In multilingual dialogue, English is best for every model and non-Indo-European languages are the worst, indicating that audio encoders and training data are not yet multilingual in practice.

Load-bearing premise

The benchmark's ambiguity scores assume the synthetic audio really carries the intended intonation and pause cues; the paper validates synthetic audio only on 1,000 general dialogues, not on the ambiguity items where those cues are the whole point.

Editorial extensions

If this is right

  • If the benchmark is right, model rankings should be reported separately per dataset, since a model that tops general dialogue can still fail ambiguity (for example, GPT-4o drops from above 8 to 6.87 on ambiguity).
  • Developers get a concrete tuning target: improving perception of intonation contours, pause placement, and homophone disambiguation should move the ambiguity score before it moves general dialogue scores.
  • The cascade pattern, where Whisper-plus-LLM pipelines beat most end-to-end LALMs, implies that the audio encoder is a primary bottleneck for dialogue understanding, not the language model's reasoning.
  • The multilingual gap implies that spoken dialogue support for non-English languages likely needs dedicated audio data, not just LLM knowledge, because the same model's base abilities do not transfer to audio in those languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not isolate whether ambiguity failures come from the audio encoder or from the downstream language model; a testable extension is to feed the same LALM the audio and the correct transcript side by side and see whether it then picks the intended meaning.
  • An implication left implicit is that the ambiguity protocol could be turned into a generation benchmark for synthetic speech evaluation: if TTS systems are graded by whether listeners recover the intended intonation-based meaning, SSML-style control becomes directly measurable.
  • The real-versus-synthetic validation is run on 1,000 general dialogues; running the same ablation on the ambiguity subset would show whether low scores are phonetic failures or artifacts of synthetic prosody, which is the paper's main unverified load-bearing assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ADU-Bench, a benchmark for evaluating open-ended audio dialogue understanding of large audio-language models (LALMs). The benchmark comprises four datasets: ADU-General (12,000 items across helpful questions, daily questions, and daily statements), ADU-Skill (3,725 items across 12 domains), ADU-Multilingual (3,600 items in 9 languages), and ADU-Ambiguity (1,390 items covering intonation-, pause-, homophone-, and repetition-based ambiguity). The evaluation uses GPT-4 as a text-based judge that scores model responses against references, with a second scoring pass after swapping reference and response positions to mitigate position bias, plus human evaluation alignment studies. The paper benchmarks 16 LALMs and reports that existing systems struggle with mathematical symbols and formulas, human behavior such as roleplay, multilingual comprehension, and ambiguity arising from phonetic elements.

Significance. If the ambiguity results are valid, ADU-Bench fills a real gap in LALM evaluation, providing a broad and much-needed instrument with over 20,000 open-ended audio dialogues. The strengths are the scale and diversity of the data, the systematic treatment of position bias through double scoring, the inclusion of multiple LLM judges and human-alignment checks, and the public release of code and data. The ambiguity dataset, however, is the paper's principal novelty and carries the main risk: its synthetic prosody is not quantitatively validated, and the GPT-4 judge never hears the audio, which weakens the causal claim that low ambiguity scores reflect failures of audio understanding rather than TTS artifacts or evaluation blind spots.

major comments (5)
  1. [Section 3.2 and Appendix A] The construction of the ADU-Ambiguity dataset relies on synthetic audio generated with SSML <prosody> and <break> tags, and Appendix A asserts that 'a manual validation process' was conducted, but no numbers are reported: there is no sample size, no accuracy, and no inter-annotator agreement. Because the entire ambiguity claim presupposes that the synthesized intonation and pause cues are correctly perceived by human listeners as the intended interpretation, the paper should report a perceptual study in which listeners hear only the audio and select the intended meaning, with agreement rates per ambiguity type. Without such validation, the low model scores on intonation- and pause-based items could be a TTS artifact rather than a genuine limitation of LALM audio understanding.
  2. [Section 5.4] The real-versus-synthetic ablation is run on 1,000 randomly sampled ADU-General dialogues only ('randomly sample 1,000 real-world audio dialogues and generate synthetic audio from their transcriptions'). This provides no evidence for the claim that 'both real-world audio and synthetic audio can effectively serve as evaluation sources' for the ADU-Ambiguity dataset, where synthetic prosody is the load-bearing variable. An analogous validation for ambiguity items, or at least a comparison of real and synthetic renditions of the same ambiguous sentences, is needed to support the paper's conclusions about phonetic ambiguity.
  3. [Section 4] The GPT-4 evaluator receives only the textual transcription of the audio query, which is identical for both interpretations of an ambiguity item. Consequently, the judge cannot verify that the reference corresponds to the prosody actually present in the audio, and responses that hedge by listing multiple interpretations may be over-scored as helpful while responses that commit to the wrong interpretation may be under-scored. This weakens the claim that the scores specifically measure audio understanding for ambiguity. The paper should validate the ambiguity scores against human raters who listen to the audio and judge whether the model response matches the intended meaning, rather than only reporting overall pairwise consistency on a small sample; Table 3's ADU-Ambiguity row is based on only 20 queries and uses pairwise preference, which does not test this audio-reference consistency.
  4. [Section 3.2 and Table 1] The number of ADU-Skill dialogues is inconsistent: Section 3.2 states the dataset comprises 3,750 audio dialogues, while Table 1 reports 3,725, and the overall total of 20,715 is consistent with 3,725. The discrepancy should be corrected and the totals cross-checked.
  5. [Abstract and Section 3.2] The paper claims to 'firstly propose the evaluation of ambiguity handling in audio dialogues' (Abstract) and 'firstly analyze the ambiguity within audio dialogues' (Section 3.2). This claim of firstness is contradicted by the cited SD-Eval (Ao et al., 2024), which explicitly evaluates spoken dialogue understanding beyond words, including phenomena related to prosody and intonation. The authors should either remove the firstness claim or clearly delineate the specific difference between ADU-Ambiguity and SD-Eval's coverage.
minor comments (5)
  1. [Figure 2] The rendered figure in the PDF appears to contain duplicated subfigures, repeated 'strategy' axis labels, and 'Loading [MathJax]' artifacts; the final figure should be cleaned and each panel should be labeled and referenced unambiguously.
  2. [Section 5.2] The sentence 'BLSP stands out with the highest average score of 3.85 among all LALMs' is ambiguous because Step-Audio-Chat (5.21) and the cascaded models score higher; the authors should specify that this is among the open-sourced end-to-end LALMs or otherwise qualify the comparison.
  3. [Table 3] The caption 'Association between human judgment and each dataset in ADU-Bench of GPT-4 evaluation' is unclear; the columns are model-pair comparisons, so the caption should state that each entry is the percentage of pairwise judgments in which the GPT-4 evaluator agrees with human preference for that model pair and dataset.
  4. [Limitations] The Limitations section mentions only the limited number of LALMs; given the load-bearing role of synthetic prosody in the ambiguity dataset, the limitations should also acknowledge the need for perceptual validation of the synthesized ambiguity cues.
  5. [Appendix D] In the human evaluation details, the sentence 'we carefully consider the ethical aspects and potential risks associated with the research involving human subjects' begins with a lowercase 'we' after a previous sentence ending in a period; this capitalization and sentence flow should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ADU-Bench is an evaluation instrument whose central claims are supported by human evaluation and independent judges; the synthetic-prosody concern is a validity risk, not a circular reduction.

full rationale

ADU-Bench is a benchmark and evaluation study rather than a derivation: it contains no fitted parameters, no equations, and no predicted quantity that is defined in terms of its own outputs. The only arguable same-family contamination is that GPT-4 is used both to generate textual references and to score LALM responses (Section 4 and Appendix A), which could in principle favor GPT-4-family behavior; however, the paper mitigates this with a human pairwise evaluation reporting above 85% agreement (Section 5.4, Fig. 3(b)), with a 140-sample direct human rating (Appendix D), and with two additional independent LLM judges (LLaMA-3-70B-Instruct and Qwen-2-72B-Instruct, Appendix F) that reproduce the main ranking conclusions. None of the paper's self-citations is load-bearing: the authors' prior works appear only in related-work lists and in the ethics acknowledgment. The weakest step is the ADU-Ambiguity synthetic audio: Appendix A states that intonation items use SSML <prosody> tags and pause items use <break> tags, and that manual validation confirmed the intended ambiguity was conveyed, but no numbers, sample size, or inter-annotator agreement are reported, and the Section 5.4 real-vs-synthetic ablation covers only 1,000 general dialogues rather than the ambiguity items. That is an external-validity and reproducibility threat, not circularity: low scores on synthetic items would be an artifact of text-to-speech rendering rather than a demonstration that the model's behavior is equivalent to the benchmark's construction. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The evaluation rests on three assumptions: text-based judging captures audio understanding, synthetic prosody carries real ambiguity, and GPT-4 scores align with humans. No free parameters are fitted, and no new physical or model entities are introduced.

assumptions (3)
  • domain assumption Textual transcriptions plus GPT-4 references are a sufficient proxy for audio understanding in evaluation.
    Sections 4 and E.2 explicitly state that evaluation uses textual transcriptions of audio and references, focusing on understanding rather than generation. This assumption is load-bearing for all four datasets.
  • domain assumption Synthetic SSML audio preserves intended ambiguity cues and is equivalent to real-world audio for benchmarking.
    Appendix A describes SSML prosody and break tags for intonation and pause ambiguity. Section 5.4 validates real versus synthetic audio only on 1,000 general dialogues, not on ambiguity items.
  • domain assumption GPT-4 scoring with swapped positions approximates human judgment of response quality.
    Section 5.4 and Appendix D report pairwise consistency above 85 percent on 140 samples, but no statistical significance or inter-annotator agreement metrics are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models." pith.science (2026). https://pith.science/paper/6BBDULYJ

@misc{pith2026241205167,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6BBDULYJ}},
  note         = {Machine review of arXiv:2412.05167}
}
read the original abstract

Large Audio-Language Models (LALMs), such as GPT-4o, have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans. The potential of LALMs broadens their applicability across a wide range of practical scenarios supported by audio dialogues. However, given these advancements, a comprehensive benchmark to evaluate the performance of LALMs in the open-ended audio dialogue understanding remains absent currently. To address this gap, we propose an Audio Dialogue Understanding Benchmark (ADU-Bench), which consists of 4 benchmark datasets. They assess the open-ended audio dialogue ability for LALMs in 3 general scenarios, 12 skills, 9 multilingual languages, and 4 categories of ambiguity handling. Notably, we firstly propose the evaluation of ambiguity handling in audio dialogues that expresses different intentions beyond the same literal meaning of sentences, e.g., "Really!?" with different intonations. In summary, ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs. Through extensive experiments on 16 LALMs, our analysis reveals that existing LALMs struggle with mathematical symbols and formulas, understanding human behavior such as roleplay, comprehending multiple languages, and handling audio dialogue ambiguities from different phonetic elements, such as intonations, pause positions, and homophones. The benchmark is available at https://adu-bench.github.io/.

Figures

Figures reproduced from arXiv: 2412.05167 by the authors.

Figure 2
Figure 2. The average scores across each domain for 4 datasets within ADU-Bench under 16 LALMs. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Ablation study on ADU-Bench. (a) Real-world and synthetic audio can both serve as evaluation sources. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. The evaluation method in ADU-Bench. To benchmark open-ended audio dialogue understanding for [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models

    eess.AS 2025-05 conditional novelty 6.0 of 10

    Most speech-aware language models follow written output-format instructions far worse than their text-only base LLMs, and Speech-IFEval measures this as catastrophic forgetting.

  2. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  3. Audio-Language Models for Audio-Centric Tasks: A Systematic Survey

    cs.SD 2025-01 conditional novelty 5.0 of 10

    A systematic survey that categorizes audio-language models by architecture, training objective, and application, covering speech, music, and general audio.

  4. Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages · cited by 4 Pith papers

  1. [2]

    We appreciate your time and effort in partici- pating in this study

    Based on the criteria of helpfulness, relevance, accuracy, and comprehensiveness, provide an over- all score on a scale of 0 to 10 for the response, where a higher score indicates better overall per- formance. We appreciate your time and effort in partici- pating in this study. Your valuable insights will significantly contribute to the development and im...

    work page

  2. [3]

    A is better

    Based on the criteria of helpfulness, relevance, accuracy, and comprehensiveness, indicate your preference. You can choose from the following options: “A is better”, “B is better”, or “Both are equal”. We appreciate your time and effort in partici- pating in this study. Your valuable insights will significantly contribute to the development and im- provem...

    work page 2024

  3. [5]

    arXiv preprint arXiv:2410.19168

    Mmau: A massive multi-task audio under- standing and reasoning benchmark. arXiv preprint arXiv:2410.19168. Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan Sharma, Wei-Lun Wu, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe

    arXiv

  4. [10]

    Listen to the audio queries carefully

    work page

  5. [11]

    Compare the two textual responses provided for each audio

    work page

  6. [1997]

    Subsequently, ex- ecute the program code using Python interpreter with public SSML service (Microsoft, 2024) pro- vided by Microsoft Azure to convert them into au- dio dialogues

    by human coding, where SSML is an XML- based markup language specifically designed for speech synthesis applications. Subsequently, ex- ecute the program code using Python interpreter with public SSML service (Microsoft, 2024) pro- vided by Microsoft Azure to convert them into au- dio dialogues. Furthermore, to emulate real-world scenarios, we consider a ...

    work page 2024

  7. [2020]

    arXiv preprint arXiv:2009.03300

    Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In NeurIPS. Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei ...

    arXiv 2009

  8. [2022]

    arXiv preprint arXiv:2212.10525

    Slue phase-2: A benchmark suite of diverse spoken language understanding tasks. arXiv preprint arXiv:2212.10525. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma...

    arXiv 2023

Show all 11 references
  1. [2023]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale au...

    2023 arXiv

  2. [2024]

    arXiv preprint arXiv:2402.12786

    Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786. Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. Webglm: Towards an eff...

    2023 arXiv

  3. [2025]

    arXiv preprint arXiv:2505.17601

    Wolf hidden in sheep’s conversations: To- ward harmless data-based backdoor attacks for jail- breaking large language models. arXiv preprint arXiv:2505.17601. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel a...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.