ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models

Dexian Wang; Jialiang Tao; Junwei Zhang; Kaiwen Luo; Kun Wang; Liang Lin; Li Sun; Moayad Aloqaily; Qingsong Wen; Yibo Zhang

REVIEW 5 major objections 5 minor 2 cited by

ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ChronosAudio, a 36,000-question benchmark over 200+ hours of audio, measures a >90% performance collapse in audio LLMs when clips stretch from seconds to 10–20 minutes, traces it to attention dilution, and finds mitigations recover only hal

desk verdict A serious long-audio benchmark effort undone by undefined metrics, inconsistent data totals, and placeholder summary ground truth — the headline findings are not reproducible. read the letter →

arxiv 2601.04876 v2 pith:VGEAXDNL submitted 2026-01-08 cs.SD

Kaiwen Luo , Liang Lin , Yibo Zhang , Moayad Aloqaily , Jialiang Tao , Dexian Wang , Zhenhong Zhou , Junwei Zhang

show 3 more authors

Kun Wang Li Sun Qingsong Wen

This is my paper · ORCID

classification cs.SD

keywords long-audiobenchmarkaudiolargelanguagemodelslengthgeneralizationattentiondilutionlostinthemiddlelong-contextevaluationspeechtranscriptionLLM-as-a-judge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds ChronosAudio, a benchmark of 36,000 test questions over 200 hours of audio, organized into six tasks and three duration bands (short, middle, long), to measure whether audio large language models understand anything beyond short clips. The central claim is that they do not: moving from short clips to 10–20 minute recordings cuts performance by over 90% on some tasks, with most of the loss already visible at 5–10 minutes. The paper attributes this collapse to structural attention dilution—attention maps lose their sharp local focus as the sequence progresses—and shows that sparse or sliding-window attention recovers at most about 50% of short-form skill. A sympathetic reader would care because long-form audio (meetings, calls, lectures) is exactly where these models would be useful, and the benchmark makes the failure measurable and comparable.

What carries the argument

The load-bearing instrument is ChronosAudio itself: 6,000 distinct audio clips (3,000 short at 30s–5min, 2,000 middle at 5–10min, 1,000 long at 10–20min) rendered into 36,000 test instances across six tasks (Dictation, Localization, Transcription, Multi-Speaker, Comprehension, Summary). The three findings are produced by length-stratified scoring on those tasks, visualization of self-attention weights across First/Middle/Last sequence segments to reveal structural attention dilution, and surgical modification of attention (Sparse Attention and Sliding Window Attention) to test the restorative ceiling. The named mechanism, structural attention dilution, is the loss of a sharp diagonal attenti

What would settle it

Re-score a stratified sample of ChronosAudio instances with concretely specified coefficients and human-verified references for the Summary task, then re-measure the short-to-long degradation; if the >90% collapse shrinks to a modest drop (or vanishes) on that sample, the paper's headline finding fails. A second check: compute a diagonal-dominance index of attention maps across sequence positions; if the index does not decline with position in models that nonetheless show the collapse, the attention-dilution mechanism is not the cause.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is that current audio large language models, both open- and closed-source, have no reliable long-audio understanding. On ChronosAudio, open-source models fall from an average of 28.06 to 0.00 on long-form transcription, and closed-source models fall from 42.66 to 3.86; comprehension for open-source models collapses to 3.90. The paper further argues that the mechanism is structural attention dilution, visible as the loss of the diagonal attention pattern in later sequence positions, and that mitigation by sparse attention or sliding windows is capped at roughly 50% of the short-context score for fidelity-heavy tasks. The authors present this as evidence that

Load-bearing premise

The numbers carry the argument: every composite score in Equations (6)–(9) depends on balancing coefficients (β1, β2, γ1, γ2, w1, w2, δ1, δ2) that the paper never assigns, and the Summary ground truth in the appendix is the placeholder 'No specific summary answer of this content'; if those coefficients are arbitrary or those labels are invalid, the 90% collapse, attention-dilution, and 50%-ceiling findings are uninterpretable.

Editorial extensions

If this is right

Long-form dictation, localization, transcription, and multi-speaker tasks are effectively unsolved: most models score near zero on 10–20 minute audio.
The lost-in-the-middle effect starts earlier in audio LLMs than in text LLMs: the steepest drop is from short to middle duration, suggesting the effective high-fidelity context window is under 5 minutes.
Attention dilution gives a concrete target for future work: models need mechanisms that preserve temporal locality across long sequences, not just longer context windows.
Sparse attention can nearly eliminate the drop on retrieval-oriented tasks (93% recovery in dictation) but leaves transcription at about 50% of short-form proficiency, so retrieval and verbatim fidelity need different fixes.
Closed-source models retain substantial high-level reasoning on long audio (Summary around 59) while open-source models collapse to about 14, indicating a gap between perception and reasoning abilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the metric coefficients in Equations (6)–(9) are never assigned, and the Summary ground truth in the appendix reads 'No specific summary answer of this content,' a re-scoring with explicit coefficients and human-validated summaries is a necessary check before the quantitative scale of the collapse is taken at face value.
If attention dilution is causal rather than merely correlational, then interventions that force local attention—chunk-wise cross-attention, audio-native positional encodings, or token downsampling—should be compared against the 50% ceiling the paper reports; a method that breaks the ceiling would validate the mechanism.
The benchmark's duration stratification enables a simple testable prediction: model performance on any fidelity task should be a monotone decreasing function of duration, so plotting score versus duration on a held-out sample would let the community check the collapse curve outside the 16 models tested.
The paper's English-only, clean-audio scope leaves open whether the collapse is a general audio-length effect or partly an artifact of the test distribution; extending the same protocol to noisy and multilingual long audio would separate those factors.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

The short version: this paper is a serious attempt at a long-audio benchmark, and the authors clearly put in the hours — 16 models, six tasks, three duration buckets, qualitative failure-mode case studies. The task taxonomy (perception, verbatim generation, reasoning) is sensible, and the qualitative examples of models refusing or looping on long inputs are genuinely illustrative. But the central empirical claims — the 90% collapse, the attention-dilution explanation, and the 50% mitigation ceiling — rest on Table 2, and Table 2 is not reproducible.

The problems are load-bearing, not cosmetic. Equations (6)–(9) define composite scores with balancing coefficients β1, β2, γ1, γ2, w1, w2, δ1, δ2, and the paper never assigns values to any of them. In the appendix, the authors describe a different procedure: three LLM judges score each answer 1–100 and the final score is their average. No bridge between those two protocols is given. You cannot tell what the numbers in Table 2 mean. The Summary task is worse: the case study labels three ground truths as 'No specific summary answer of this content.' There is no G to plug into Eq. (9). If the judge is comparing against the full transcript, that's a different metric than the one advertised. The limitations section mentions English-centric data and the lack of training-based fixes, but it doesn't acknowledge the metric problem.

The dataset statistics also don't add up. 6,000 clips × 322 seconds ≈ 537 hours, not the 'over 200 hours' in the abstract or the 237h in Table 1. And the 'first long-audio benchmark' claim is directly contradicted by BLAB, which appears in their own Table 1 with an 833h, 51-minute average.

What's genuinely new is the benchmark artifact itself — 36,000 instances across six task types stratified by duration. If the authors fix the metrics, release a data card, correct the volume accounting, and recalibrate their novelty claims, this could be a useful resource. As it stands, the headline findings are unverifiable and the paper needs major revision before it is referee-ready. I would not cite it in its current form.

Referee Report

5 major / 5 minor

Summary. The paper introduces ChronosAudio, a proposed multi-task benchmark for evaluating audio large language models (ALLMs) on long-form audio. It contains 36,000 test instances over six tasks (Dictation, Localization, Transcription, Multi-Speaker, Comprehension, Summary), stratified into short, middle, and long durations, and reports evaluations of 16 open- and closed-source models. The central claims are: (1) ALLMs show a 'precipitous' performance collapse of over 90% when moving from short to long audio; (2) this collapse is caused by 'Structural Attention Dilution', a diffusion of attention in later sequence positions; and (3) existing mitigation strategies such as sparse attention recover only about 50% of short-context proficiency. The paper also proposes mitigation experiments with sparse and sliding-window attention.

Significance. If the benchmark and its measurements were valid, this would be a useful contribution to an important and underexplored problem: how ALLMs behave on minute-scale and document-scale audio. The paper has strengths: it covers 16 models, spans six task types, stratifies by duration, and releases code/data. The appendix includes raw prompt/response examples and a transparent ensemble-LLM-judge protocol, which is commendable. However, the central empirical findings are not currently supported: the composite metrics in Eqs. (6)–(9) are under-specified, the Summary ground truth appears to be a placeholder, the data volume is internally inconsistent, and the attention-dilution explanation rests on a qualitative visualization of a single model. These issues compromise the interpretability of every headline number in Table 2, and therefore the significance of the paper's conclusions.

major comments (5)

[§3.3–3.4, Eqs. (6)–(9)] The composite scores for Transcription, Multi-Speaker, Comprehension, and Summary are not computable as written. The balancing coefficients β1, β2, γ1, γ2, w1, w2, δ1, δ2 are never assigned values, and no constraints such as β1+β2=1 are stated. Appendix 3 then describes a completely different protocol: three LLM judges each give an integer 1–100 and the final score is the arithmetic mean. The relationship between this judge-based score and Eqs. (6)–(9) is never explained. Since all downstream observations—including the >90% collapse, the attention-dilution claim, and the 50% recovery ceiling—rely on the resulting Table 2 scores, the central empirical claims are not reproducible.
[§3.1 and Table 1] The reported scale of the benchmark is internally inconsistent. Section 3.1 states 6,000 distinct audio clips with an average length of 322 seconds. That product is 6,000 × 322 s ≈ 537 hours, but the abstract claims 'over 200 hours' and Table 1 reports '237h'. The paper must reconcile these numbers; as it stands, the reader cannot tell how much audio the benchmark actually contains, which affects every claim about long-form coverage and model workload.
[Appendix, Summary Task case study] The ground-truth labels for the Summary task are shown as 'No specific summary answer of this content' for the Short, Middle, and Long examples. Equation (9) requires a ground-truth key-point set K(G) to compute coverage and factuality, but no such summary reference is provided. Moreover, the LLM-judge prompt in Figure 6 supplies the 'Audio Transcript (Ground Truth)', not a reference summary. Thus the Summary scores in Table 2—including the open-source 40.96→14.14 and closed-source 75.00→59.22 comparisons—are not valid measurements of summarization quality as defined by Eq. (9).
[§4.3 and Figure 4] The 'Structural Attention Dilution' explanation is not established. The evidence consists of qualitative attention heatmaps for a single model, Qwen2-Audio-7B, at three 19-token windows marked First, Middle, and Last. No quantitative definition of 'dilution' is given, no entropy or diagonal-concentration metric is reported, no comparison is made to models that do not collapse, and no link is shown between the heatmap pattern and the task scores in Table 2. As presented, this is an illustration, not a causal explanation.
[§4.4, Obs.❻] The '50% recovery' ceiling is based on a single example: Qwen2.5-Omni-3B Transcription, where Sparse Attention gives 25.20 versus the short-form baseline of 50.10. Table 3 reports only three open-source models, and despite the claim in Table 2 that scores are 'averaged over 5 experimental rounds', no variance or significance is reported. Generalizing from one model–task pair to a universal 'restorative ceiling' is unsupported.

minor comments (5)

[§4.2, Obs.❶] The text 'with a remarkable decline about 18 points, from 3.37 to 21.30 ↑' is garbled: the numbers appear in the wrong order and the arrow direction is inconsistent with a decline. Please correct.
[Eq. (8)] The symbol I(·) is introduced in §3.4 as an 'information extraction function' but is then used in Eq. (8) as an indicator function. This ambiguity makes the comprehension metric hard to interpret.
[Appendix, Summary case study headers] The case-study headings for the Summary task are mislabeled as 'Comprehension Task (Middle)' and 'Comprehension Task (Long)', even though the prompts and labels are for summarization. This is confusing and should be fixed.
[References] References 'Li et al. 2025a' and 'Li et al. 2025b' appear to be the same paper (ISA-Bench). Please merge or distinguish them.
[Table 1] The formatting of Table 1 is broken: benchmark names are concatenated with durations (e.g., '400h14s'), making the comparison difficult to parse. A clean table is needed.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the benchmark findings are empirical measurements, not reductions to their own inputs; the self-citation is minor and non-load-bearing. Metric underspecification and missing Summary gold labels are correctness risks, not circularity.

full rationale

The paper's claimed chain is empirical rather than derivational: Eqs. (4)-(9) define per-task metrics, Table 2 reports measured scores, Figure 4 visualizes attention weights, and Table 3 compares attention variants against short-form baselines. None of the headline findings reduces to a fitted parameter or to the definition of the thing being predicted. The >90% collapse is an arithmetic comparison of Table 2 values (e.g., closed-source Transcription 42.66 to 3.86); the 50% ceiling is a ratio of a recovered middle-form score to a short-form baseline (25.20/50.10 for Qwen2.5-Omni-3B Transcription); and attention dilution is read off attention heatmaps, not derived from the performance metric. The only overlap with prior work by the same authors is Lin et al. (2025), cited in Section 1 in a list of datasets/benchmarks, so it is not load-bearing. The Appendix's LLM-as-a-judge protocol is an external measurement procedure (three text-side LLMs scoring against the transcript), not a self-referential definition. However, two manuscript-internal flags must be recorded as validity problems, not circularity: (i) balancing coefficients beta1, beta2, gamma1, gamma2, w1, w2, delta1, delta2 in Eqs. (6)-(9) are never assigned values, and Appendix 3 substitutes a judge mean for the advertised formulas; (ii) the Appendix Case Study labels Summary ground truth as 'No specific summary answer of this content,' so Eq. (9) has no defined K(G). These make Table 2 difficult to reproduce or interpret, but they do not exhibit the specific kind of reduction (Eq. X = Eq. Y by construction, fitted parameter renamed as prediction) required for a circularity finding.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity depends on several unstated or unjustified choices: the balancing coefficients in all composite metrics are never given values, so the reported scores are underdetermined; the Summary task has no real reference summary ("No specific summary answer"); the audio provenance (apparently TTS-generated from text) is never described; and the attention-dilution conclusion rests on three small hand-picked heatmap windows. No free parameters are fitted to data, but the unset weights act as hidden free parameters that could change every ranking in Table 2.

free parameters (5)

β1, β2 (Transcription weights)
Eq. (6) defines TS as a weighted sum but the values are never specified; Table 2 scores depend on them.
γ1, γ2 (Multi-Speaker weights)
Eq. (7) similarly leaves the balancing coefficients unspecified.
w1, w2 (Comprehension weights)
Eq. (8) uses weights for exact match vs hallucination penalty; never set.
δ1, δ2 (Summary weights)
Eq. (9) uses weights for coverage/factuality; never set.
τ (Localization tolerance) = 0.1s
Strict tolerance threshold in Eq. (5) chosen by hand; no sensitivity analysis is given.

assumptions (5)

domain assumption Ground-truth labels for the Summary task are valid references
Appendix Case Study shows the label "No specific summary answer of this content," implying there is no reference summary; scoring against it is meaningless.
domain assumption LLM-as-a-judge scores are a valid proxy for human quality
Appendix 3 claims "recent studies verify" correlation with human judgments but provides no citation; the three judges' arithmetic mean is used as the ground truth for Summary/Comprehension.
ad hoc to paper Audio generated from text (apparently TTS) is representative of real long-form audio
No data provenance section; prompts and labels suggest synthetic audio from text, but the paper never states how the audio was created.
ad hoc to paper Attention heatmap patches (first/middle/last 19-token windows) are representative of global attention behavior
Figure 4 visualizes only three small windows at arbitrary token indices; no quantitative attention metric is reported.
ad hoc to paper The balancing weights are implicitly fixed (e.g., equal) in reported tables
Since weights are unspecified, the reported scores must assume some implicit weighting; the paper does not state this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models." pith.science (2026). https://pith.science/paper/VGEAXDNL

@misc{pith2026260104876,
  author       = {Pith},
  title        = {Pith review of: ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGEAXDNL}},
  note         = {Machine review of arXiv:2601.04876}
}

read the original abstract

Although Audio Large Language Models (ALLMs) have witnessed substantial advancements, their long audio understanding capabilities remain unexplored. A plethora of benchmarks have been proposed for general audio tasks, they predominantly focus on short-form clips, leaving without a consensus on evaluating ALLMs over extended durations. This paper proposes ChronosAudio, the first multi-task benchmark tailored for long-audio understanding in ALLMs. It encompasses six major task categories and comprises 36,000 test instances totaling over 200 hours audio, stratified into short, middle, and long-form categories to comprehensively evaluate length generalization. Extensive experiments on 16 state-of-the-art models using ChronosAudio yield three critical findings: 1.Precipitous Long-Context Collapse: ALLMs exhibit a severe inability to sustain performance, with the transition from short to long contexts triggering a staggering performance degradation of over 90% in specific tasks. 2.Structural Attention Dilution: Performance degradation stems from a fundamental failure in maintaining temporal locality; attention mechanisms suffer from significant diffusion in later sequences. 3.Restorative Ceiling of Mitigation: Current strategies only offer 50% recovery. These findings reveal significant challenges in long-audio, underscoring the urgent need for approaches to achieve robust, document-level audio reasoning.

Figures

Figures reproduced from arXiv: 2601.04876 by the authors.

**Figure 2.** Overview of the ChronosAudio benchmark data composition. Each task comprises 6,000 test instances, totaling 36,000 samples. context benchmarks, ChronosAudio explicitly focuses on time-extended audio scenarios. The benchmark aggregates a diverse collection of audio data, totally amounting to over 200 hours. Specifically, ChronosAudio comprises 36,000 test instances derived from 6,000 distinct audio clips. These ins… view at source ↗

**Figure 3.** Average Performance between Closed-Source and Open-Source Models across Audio Durations. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Visualization of Attention Consistency. The heatmaps display the self-attention weights for the First, Middle, and Last segments. Darker colors represent lower weights, while brighter colors indicate higher attention scores. duration sequences, we visualize the attenti…

**Figure 5.** Figure 5: A visualization of the long audio case study on evaluated SOTA ALLMs. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: The prompt template used for the Ensemble LLM-as-a-Judge framework. To ensure a robust and unbiased evaluation, we employ three diverse state-of-the-art LLMs, DeepSeek-V3.1-Terminus, Qwen3-VL-235BA22B-Thinking, and Kimi-K2-Thinking as independent judges. Each model re…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing
cs.CL 2026-07 conditional novelty 7.0 of 10

REDDIT corrects non-speech-induced timestamp drift in autoregressive ASR by editing timestamp targets under cached replay context while anchoring non-timestamp behavior to the frozen base distribution.
EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs
cs.CL 2026-05 unverdicted novelty 6.0 of 10

EchoDistill applies noisy-to-clean self-distillation with GRPO to boost Audio LLM robustness, reporting 4.18% average GSR gains under strong noise.

Reference graph

Works this paper leans on

10 extracted references · 4 linked inside Pith · cited by 2 Pith papers

[1]

Audio Transcript (Ground Truth): {transcript}
[2]

Ziyang Ma and 1 others

Auditing meta-cognitive hallucinations in reasoning large language models.arXiv preprint arXiv:2505.13143. Ziyang Ma and 1 others. 2025. Mmar: A challenging benchmark for deep reasoning in speech, audio, mu- sic, and their mix.arXiv preprint arXiv:2505.13032. Zeping Min and Jinbo Wang. 2023. Exploring the in- tegration of large language models into automa...

arXiv 2025
[3]

Zhe Sun, Yujun Cai, Jiayu Yao, and Yiwei Wang

Audio jailbreak: An open comprehensive benchmark for jailbreaking large audio-language models.arXiv preprint arXiv:2505.15406. Zhe Sun, Yujun Cai, Jiayu Yao, and Yiwei Wang

arXiv
[4]

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang

Spatial blind spot: Auditory motion per- ception deficits in audio llms.arXiv preprint arXiv:2511.13273. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289. Jinchuan Tian, Sang-gil Lee, Zhifeng Kon...

arXiv 2023
[6]

Qian Yang and 1 others

Qwen3-omni technical report.arXiv preprint arXiv:2509.17765. Qian Yang and 1 others. 2024. Air-bench: A compre- hensive benchmark for audio-image-text reasoning. arXiv preprint arXiv:2402.07729. Zonghan Yang, Fang Xiong, Peng Li, and Yang Liu

arXiv 2024
[7]

Listen to the audio corresponding to the given text, and what is the last word spoken in the audio?

Rethinking long context generation from the continual learning perspective. InProceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 1922–1933. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on ...

arXiv 2025
[9]

User Question: {question}
[10]

prompt":

Model Generated Response: {response} [Evaluation Output] Please output your judgment in the following format: 1.Analysis:Provide a concise analysis of the pros and cons of the response. 2.Score:Give a final integer score from 1 to 100 (where 100 is perfect), according to the fomula of each task. Figure 6:The prompt template used for the Ensemble LLM-as-a-...

1916

Show all 10 references

[2024]

InPro- ceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR)

Muchomusic: Evaluating music understand- ing in multimodal audio-language models. InPro- ceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR). Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang ...

2025 arXiv
[2025]

Haolang Lu, Yilian Liu, Jingxin Xu, Guoshun Nan, Yuanlong Yu, Zhican Chen, and Kun Wang

Tta: Transcribe, translate and alignment for cross-lingual speech representation.arXiv preprint arXiv:2511.14410. Haolang Lu, Yilian Liu, Jingxin Xu, Guoshun Nan, Yuanlong Yu, Zhican Chen, and Kun Wang

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Audio Transcript (Ground Truth): {transcript}

[2] [2]

Ziyang Ma and 1 others

Auditing meta-cognitive hallucinations in reasoning large language models.arXiv preprint arXiv:2505.13143. Ziyang Ma and 1 others. 2025. Mmar: A challenging benchmark for deep reasoning in speech, audio, mu- sic, and their mix.arXiv preprint arXiv:2505.13032. Zeping Min and Jinbo Wang. 2023. Exploring the in- tegration of large language models into automa...

arXiv 2025

[3] [3]

Zhe Sun, Yujun Cai, Jiayu Yao, and Yiwei Wang

Audio jailbreak: An open comprehensive benchmark for jailbreaking large audio-language models.arXiv preprint arXiv:2505.15406. Zhe Sun, Yujun Cai, Jiayu Yao, and Yiwei Wang

arXiv

[4] [4]

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang

Spatial blind spot: Auditory motion per- ception deficits in audio llms.arXiv preprint arXiv:2511.13273. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289. Jinchuan Tian, Sang-gil Lee, Zhifeng Kon...

arXiv 2023

[5] [6]

Qian Yang and 1 others

Qwen3-omni technical report.arXiv preprint arXiv:2509.17765. Qian Yang and 1 others. 2024. Air-bench: A compre- hensive benchmark for audio-image-text reasoning. arXiv preprint arXiv:2402.07729. Zonghan Yang, Fang Xiong, Peng Li, and Yang Liu

arXiv 2024

[6] [7]

Listen to the audio corresponding to the given text, and what is the last word spoken in the audio?

Rethinking long context generation from the continual learning perspective. InProceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 1922–1933. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on ...

arXiv 2025

[7] [9]

User Question: {question}

[8] [10]

prompt":

Model Generated Response: {response} [Evaluation Output] Please output your judgment in the following format: 1.Analysis:Provide a concise analysis of the pros and cons of the response. 2.Score:Give a final integer score from 1 to 100 (where 100 is perfect), according to the fomula of each task. Figure 6:The prompt template used for the Ensemble LLM-as-a-...

1916