Pith. sign in

REVIEW 4 major objections 5 minor 233 references

A frozen text teacher can train an audio-language model to reason better from sound.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:05 UTC pith:ZJMRNHJB

load-bearing objection A useful extension of on-policy distillation to audio with a genuinely new three-tier corpus, but the claim of improving audio-grounded reasoning is only partially supported—the gains are concentrated on text-recoverable tasks and the teacher's text-blindness is a real limitation. the 4 major comments →

arxiv 2607.21550 v1 pith:ZJMRNHJB submitted 2026-07-23 cs.LG

X³-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

Dongjie Fu , Di Cao , Xize Cheng , Zihan Zhang , Wenxu Jia , Yifu Chen , Shengpeng Ji , Yu Zhang
show 1 more author
Tao Jin
This is my paper
classification cs.LG
keywords cross-modal distillationon-policy alignmentaudio-language modelschain-of-thought reasoningknowledge transferparalinguistic cuesacoustic reasoningthree-tier symmetric corpus
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that cross-modal on-policy distillation can close much of the reasoning gap between large audio-language models and text-only LLMs. Instead of having the student imitate a teacher's pre-generated reasoning traces, the student rolls out its own chains-of-thought conditioned on raw audio, and a frozen text teacher scores those exact tokens using a matched text input plus the verified answer. The token-level advantage signal pushes the student toward sound-grounded, logically valid steps at the states the student actually visits. On audio-reasoning benchmarks the method raises BIG Bench Audio from 87.9 to 93.6, improves MMSU semantics, MMAU sound/speech, and MMAR chain-of-thought quality, while barely disturbing out-of-domain capabilities.

Core claim

The central claim is that reasoning ability transfers across the text-audio boundary most effectively when supervision is anchored to the student's own acoustic-conditioned rollouts. For each symmetric sample (xt, xa, q, a*), the student samples K trajectories from πS(·|xa,q); the teacher logs per-token probabilities under πT(·|xt,q,a*,y_<t); the difference defines a cross-modal advantage that weights a policy-gradient loss over the student's own tokens. Because the teacher sees the gold answer and the matched text input, it supplies a high-confidence reference even for open-ended questions, while the on-policy sampling ensures correction lands where the student's acoustic perception has dri

What carries the argument

The load-bearing object is the cross-modal advantage A(y_t) = log πT(y_t | xt, q, a*, y_<t) − log πS(y_t | xa, q, y_<t) (Eq. 2). It measures, token by token, how much more likely the frozen text teacher is—given the matched text, question, and gold answer—to generate the student's continuation than the student itself is, given its acoustic premise. Plugged into a policy-gradient loss (Eq. 3) with an importance ratio and K student rollouts, it turns the teacher's token-level confidence into a dense, state-dependent training signal. The supporting structure is a three-tier symmetric corpus—text reasoning rendered into speech, caption-grounded audio events, and prosody-aware dialogue—that suppl

Load-bearing premise

The teacher's text input must preserve the acoustic evidence the question depends on; if captions, transcripts, or meta-captions lose the audible cues, the token scores cannot teach the student to reason from sound.

What would settle it

Take two audio clips with identical transcripts but opposite paralinguistic cues (sincere vs. sarcastic delivery) and a question answerable only from prosody; train with the full X3-OPD recipe and test on both clips. If the student's accuracy and cited reasoning steps do not track the prosody-conditioned Tier-3 signal—or match a model trained on transcripts alone—the claimed transfer of acoustic reasoning is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • An audio-language model can gain chain-of-thought competence for acoustic reasoning without large-scale human audio-CoT annotation, provided a strong text teacher and paired text/audio data exist.
  • Offline SFT regresses on low-signal perceptual axes (the paper reports drops of 2.8–4.2 points on MMSU Phonology/Style and MMAU Music), while on-policy scoring keeps those drops under 2.6 points.
  • The three data tiers are complementary: dropping any one tier hurts a different capability—Tier-1 text logic, Tier-2 sound-event reasoning, Tier-3 prosody and conversational semantics.
  • Warm-start duration is decisive: no warm-start adds variance and costs 1.9–3.1 points, while a full-corpus warm-start overfits and collapses MMAU-Music by 4.2 points.
  • Out-of-domain capability is largely preserved: audio-visual and video benchmarks lose only 1.2–2.3 points with the proposed method, versus 5.3–6.6 points for offline baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method's ceiling is set by how faithfully xt represents xa. On tasks where the decisive evidence lives in timbre, micro-prosody, or music, a caption-based teacher cannot score the student meaningfully; expect the residual Phonology/Style/Music gaps to persist until an audio-grounded reward replaces the text teacher.
  • The same advantage formula could transfer to other asymmetric modality pairs, such as video-to-text reasoning, whenever a matched caption or transcript can anchor a student's perceptual rollouts.
  • A testable extension: build two audio clips with identical transcripts but opposite paralinguistic cues (e.g., sincere versus sarcastic delivery) and a question only the prosody can answer. If X3-OPD training does not change the student's answer distribution or chain-of-thought relative to a transcript-only model, the teacher signal is not carrying acoustic evidence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes X3-OPD, a cross-modal on-policy distillation framework for transferring reasoning ability from a frozen text-only LLM teacher to a large audio-language student. The student rolls out reasoning trajectories conditioned on its own acoustic perception, and the teacher scores these trajectories token-by-token under a matched textual input and the gold answer, using the log-ratio advantage in Eq. (2). A three-tier symmetric corpus is introduced: text reasoning rendered to speech (Tier 1), caption-grounded audio-event reasoning (Tier 2), and prosody-aware spoken dialogue (Tier 3). The paper reports substantial gains over the base model and offline/on-policy baselines on MMSU, MMAU, BIG Bench Audio, and MMAR, while largely preserving out-of-domain capabilities. The Limitations section acknowledges that the text teacher is blind to non-semantic acoustic cues not captured in the text input.

Significance. If the central claim holds, X3-OPD offers a practical way to use powerful text-only models to improve reasoning in audio-language models without the exposure bias of offline distillation, and it extends supervision beyond plain transcripts via enriched captions/meta-captions. The paper includes controlled in-house baselines, a three-tier ablation, and CoT-quality metrics, which are valuable. However, the evidence that the gains reflect genuinely acoustic (rather than text-recoverable) reasoning is incomplete: the largest improvements are on tasks where the teacher's textual description may already carry the needed information, and the paper reports regressions on perceptual axes. The theoretical claim about reverse-KL minimization is plausible but needs a precise statement about how the gradient is computed. Overall, the contribution is solid but the headline claim outruns the current evidence.

major comments (4)
  1. [§5.1, Appendix B] The GKD baseline is ambiguously and possibly incorrectly described. It says GKD 'optimizes the student policy against the same teacher under the forward KL divergence on on-policy student rollouts.' In the standard distillation terminology, forward KL D_KL(πT||πS) requires teacher rollouts (or importance weighting from teacher), not student rollouts; on-policy student rollouts are the natural estimator for reverse KL D_KL(πS||πT). If the baseline was actually implemented as reverse KL, then it is very close to the proposed loss (Eq. 3) and the difference must be stated precisely (e.g., no answer-conditioning, no importance ratio, no K-rollout averaging). If it was implemented as forward KL with student rollouts, it is not faithful to GKD. This is load-bearing because X3-OPD's improvement over the best on-policy baseline is a central empirical claim.
  2. [§5.2, Tables 1 and 4; Abstract] The claim that the method 'extends cross-modal distillation beyond textually recoverable content to reasoning grounded in non-linguistic events, prosody, and conversational context' is not yet supported by the evidence. The largest gains are on BIG Bench Audio (+5.7) and MMSU-Semantics (+1.9), while the purely perceptual axes (MMSU-Phonology, MMSU-Style, MMAU-Music) decline by 1.0–2.5 points. The ablation in Table 4 shows that removing Tier-1 (textual reasoning rendered to speech) causes the largest BAB drop (−4.5), consistent with the possibility that much of the headline gain is text-logical rather than acoustic. The paper should report BAB per-subtask (acoustic QA, multi-event reasoning, audio-temporal grounding, paralinguistic inference) and compare against a teacher that receives only a plain transcript (without enriched caption/meta-caption) to demonstrate that the gains are not me
  3. [§4.3, Eq. (3); §5.4] The claim that Eq. (3) 'inherently minimizes the reverse-KL divergence' against the teacher is correct only if the gradient is taken through both the importance ratio and the logπS term inside A(y_t). If A is treated as a fixed reward (e.g., stop-gradient on logπS), the surrogate is biased and no longer an unbiased estimator of D_KL(πS||πT). The implementation appendix says student log-probabilities are recomputed, but it does not state whether the gradient flows through logπS in A. Please clarify the autodiff treatment. This matters because §5.4 uses the reverse-KL property to explain capability preservation.
  4. [§4.3, Eq. (2); §4.1] The method's handling of open-ended queries is unclear. Eq. (2) conditions the teacher on the gold answer a*, and Section 4.3 says the objective 'is suited for open-ended instructional data lacking gold standard answers.' If there is no gold answer, what is a*? If the teacher generates a*, this should be stated, and the effect of using teacher-generated answers rather than verified ones should be discussed. As written, the schema always includes a*, so the claimed applicability to open-ended data is not demonstrated.
minor comments (5)
  1. [§4.3] The term 'cross-modal advantage' is a misnomer: A(y_t) is a log-ratio token reward, not an advantage in the RL sense (no baseline, no value function). Please define it as a token-level reward or KL surrogate to avoid confusion.
  2. [§5.1, §4.1] Tier-2 captions are generated by Qwen3-Omni-30B-A3B-Captioner, from the same model family as the student. This introduces a partial circularity: the teacher-side text input is produced by a model closely related to the student. Please discuss this and, if feasible, include a variant using an independent captioner or human captions.
  3. [Table 2] The CRS improvement is +0.01 (0.85 to 0.86), which is within typical noise. Report variance or statistical significance for the CoT-quality metrics, and avoid overinterpreting this small difference.
  4. [Throughout] Minor typographical and formatting issues: inconsistent spelling 'X 3-OPD' vs 'X3-OPD'; 'log-probability' hyphenation; some citations lack year consistency. Please proofread.
  5. [§5.2] No standard deviations or significance tests are reported for any table. Given the small differences on some axes (e.g., MMSU-Traits +0.2), please add error bars or state that the differences are not statistically tested.

Circularity Check

0 steps flagged

No load-bearing circularity: the distillation objective uses a frozen external text teacher and is evaluated on held-out public benchmarks; minor self-citations are not load-bearing.

full rationale

The claimed derivation chain is not circular. Eq. (3) is an on-policy policy-gradient objective with advantage A(y_t) = log πT(y_t|x_t,q,a*,y_<t) − log πS(y_t|x_a,q,y_<t); the teacher πT is Qwen3-235B-A22B-Thinking, frozen and external to the student, and the gold answer a* is external supervision. No parameter is fitted to any benchmark, and the student is initialized from the existing Qwen3-Omni-Thinking model rather than from the benchmark labels. All headline results are measured on public held-out suites (MMSU, MMAU, BIG Bench Audio, MMAR), with the base model as an explicit anchor, so the empirical claim is self-contained against external benchmarks. The only author-overlapping citation used as motivation is X-OPD (Cao et al., 2026), but the present method and experiments do not reduce to it, so the self-citation is not load-bearing. The paper's own limitation that the text teacher is 'inherently blind to non-semantic acoustic cues' weakens the strongest audio-grounding interpretation, but this is an acknowledged scope/construct-validity caveat, not a definitional equivalence or fitted input renamed as prediction. Tier-2 captions are generated by a Qwen3-Omni captioner from the same model family as the student, which is a potential source of bias, but the teacher is a different, larger frozen model, and the supervision is still decoupled from benchmark outcomes; no equation or fitted constant makes the result equal to its input by construction. Therefore no circular step is exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim is an empirical training recipe, so the ledger records hand-set data/training choices and three domain assumptions. No new physical or model entities are introduced.

free parameters (3)
  • Offline warm-start size and tier ratio (12K, 5:3:2) = 12K; 5:3:2
    Chosen by hand; ablation shows a full-data warm-start collapses MMAU-Music by −4.2, so this choice is load-bearing for the capability-preservation claim.
  • Rollout count K = 4
    Chosen to trade variance against compute; affects the on-policy gradient estimator in Eq. 3.
  • Tier-1 ASR filter threshold = WER ≤ 5%
    Hand-set quality gate; determines which TTS samples enter the corpus.
axioms (3)
  • domain assumption The teacher's text input x_t (caption/transcript/meta-caption) is a sufficient representation of the acoustic evidence needed for correct reasoning.
    Used in Eq. 2/3 (§4.3); the teacher scores the student's trajectory from x_t only. The chosen text may omit non-semantic cues; the authors acknowledge this in Limitations.
  • domain assumption Teacher token-level log-probabilities conditioned on the gold answer are a valid reward signal for reasoning-quality distillation.
    Core of Eq. 2; assumes higher teacher likelihood on a token implies a more correct reasoning step, even though the teacher has access to a* and the student does not.
  • domain assumption The evaluation benchmarks measure audio-grounded reasoning and are not substantially contaminated by the training corpus.
    Required for Table 1-3 gains to transfer; no contamination analysis is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 17106 in / 15348 out tokens · 147948 ms · 2026-08-01T07:05:41.284847+00:00 · methodology

0 comments
read the original abstract

While large audio-language models have achieved remarkable progress in auditory perception, they still lag behind text-based large language models in deep logical reasoning, primarily due to the scarcity of high-quality audio reasoning data. To bridge this gap, we propose X$^3$-OPD, a cross-modal on-policy distillation framework that transfers reasoning capabilities from a powerful text teacher to an audio-language student. During training, the student generates reasoning trajectories conditioned on its own acoustic perception, while the teacher provides token-level guidance using matched textual inputs and verified answers. We further construct a three-tier symmetric corpus covering textual reasoning rendered into speech, audio-event reasoning grounded in complex acoustic scenes, and spoken-dialogue reasoning involving paralinguistic cues. This design extends cross-modal distillation beyond textually recoverable content to reasoning grounded in non-linguistic events, prosody, and conversational context. Experiments on MMSU, MMAU, BIG Bench Audio, and MMAR demonstrate that X$^3$-OPD substantially improves audio-grounded reasoning and chain-of-thought quality while largely preserving the model's existing capabilities under domain shift.

Figures

Figures reproduced from arXiv: 2607.21550 by Di Cao, Dongjie Fu, Shengpeng Ji, Tao Jin, Wenxu Jia, Xize Cheng, Yifu Chen, Yu Zhang, Zihan Zhang.

Figure 1
Figure 1. Figure 1: Overview of X 3 -OPD. Left: a three-tier symmetric corpus—Logical (text + TTS speech), Audio (audio + refined caption), and Dialogue (dialogue + prosody-aware meta-caption)—providing (xt, xa, q, a⋆ ) for every instance. Right: an offline warm-start (Loff, Eq. 1) followed by cross-modal on-policy distillation, where the student rolls out under (xa, q) and the frozen teacher scores the same trajectories unde… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

233 extracted references · 3 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  9. [9]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  10. [10]

    arXiv preprint arXiv:2507.16632 , year=

    Step-Audio 2 Technical Report , author=. arXiv preprint arXiv:2507.16632 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2412.15649 , year=

    Slam-omni: Timbre-controllable voice interaction system with single-stage training , author=. arXiv preprint arXiv:2412.15649 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2507.08128 , year=

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models , author=. arXiv preprint arXiv:2507.08128 , year=

    Pith/arXiv arXiv

  13. [13]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  14. [14]

    Alexis Plaquet and Hervé Bredin , title=

  15. [15]

    International conference on machine learning , pages=

    Robust speech recognition via large-scale weak supervision , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  16. [17]

    https://cdn.openai.com/gpt-4o-system-card.pdf , year=

    GPT-4o System Card , author=. https://cdn.openai.com/gpt-4o-system-card.pdf , year=

  17. [18]

    https://openai.com/index/chatgpt-can-now-see-hear-and-speak/ , year=

    ChatGPT can now see, hear, and speak , author=. https://openai.com/index/chatgpt-can-now-see-hear-and-speak/ , year=

  18. [19]

    2009 , publisher=

    Spoken dialogue systems , author=. 2009 , publisher=

    2009

  19. [20]

    2016 , publisher=

    Automatic speech recognition , author=. 2016 , publisher=

    2016

  20. [21]

    International Journal of Speech Technology , volume=

    Emotion recognition and adaptation in spoken dialogue systems , author=. International Journal of Speech Technology , volume=. 2010 , publisher=

    2010

  21. [22]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  22. [23]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  23. [24]

    2025 , eprint=

    VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents , author=. 2025 , eprint=

    2025

  24. [25]

    2025 , eprint=

    VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model , author=. 2025 , eprint=

    2025

  25. [26]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Audiogpt: Understanding and generating speech, music, sound, and talking head , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  26. [27]

    arXiv preprint arXiv:2402.01831 , year=

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities , author=. arXiv preprint arXiv:2402.01831 , year=

    Pith/arXiv arXiv

  27. [28]

    arXiv preprint arXiv:2305.11000 , year=

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities , author=. arXiv preprint arXiv:2305.11000 , year=

    Pith/arXiv arXiv

  28. [29]

    Acoustics, speech, and signal processing, ieee international conference on , volume=

    SWITCHBOARD: Telephone speech corpus for research and development , author=. Acoustics, speech, and signal processing, ieee international conference on , volume=. 1992 , organization=

    1992

  29. [30]

    arXiv preprint arXiv:1810.02508 , year=

    Meld: A multimodal multi-party dataset for emotion recognition in conversations , author=. arXiv preprint arXiv:1810.02508 , year=

    Pith/arXiv arXiv

  30. [31]

    , author=

    The Fisher corpus: A resource for the next generations of speech-to-text. , author=. LREC , year=

  31. [32]

    Language resources and evaluation , volume=

    IEMOCAP: Interactive emotional dyadic motion capture database , author=. Language resources and evaluation , volume=. 2008 , publisher=

    2008

  32. [33]

    Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL) , pages=

    The second dialog state tracking challenge , author=. Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL) , pages=

  33. [34]

    arXiv preprint arXiv:2308.05725 , year=

    Expresso: A benchmark and analysis of discrete expressive speech resynthesis , author=. arXiv preprint arXiv:2308.05725 , year=

    Pith/arXiv arXiv

  34. [35]

    ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Dailytalk: Spoken dialogue dataset for conversational text-to-speech , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=

    2023

  35. [36]

    arXiv preprint arXiv:2402.12786 , year=

    Advancing large language models to capture varied speaking styles and respond properly in spoken conversations , author=. arXiv preprint arXiv:2402.12786 , year=

    Pith/arXiv arXiv

  36. [37]

    Advances in Neural Information Processing Systems , volume=

    Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents , author=. Advances in Neural Information Processing Systems , volume=

  37. [38]

    arXiv preprint arXiv:2401.00475 , year=

    E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models , author=. arXiv preprint arXiv:2401.00475 , year=

    Pith/arXiv arXiv

  38. [39]

    PCWorld Retrieved October , volume=

    The Microsoft-Amazon Deal Leaves Cortana Speakers with One Advantage: Skype , author=. PCWorld Retrieved October , volume=

  39. [40]

    arXiv preprint arXiv:2502.17239 , year=

    Baichuan-audio: A unified framework for end-to-end speech interaction , author=. arXiv preprint arXiv:2502.17239 , year=

    Pith/arXiv arXiv

  40. [41]

    Medical reference services quarterly , volume=

    Alexa, Siri, Cortana, and more: an introduction to voice assistants , author=. Medical reference services quarterly , volume=. 2018 , publisher=

    2018

  41. [42]

    arXiv preprint arXiv:2312.15185 , year=

    emotion2vec: Self-supervised pre-training for speech emotion representation , author=. arXiv preprint arXiv:2312.15185 , year=

    Pith/arXiv arXiv

  42. [43]

    International Conference on Machine Learning , pages=

    BEATs: Audio Pre-Training with Acoustic Tokenizers , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  43. [44]

    Advances in Neural Information Processing Systems , year=

    Attention is all you need , author=. Advances in Neural Information Processing Systems , year=

  44. [45]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  45. [47]

    Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

  46. [48]

    Text summarization branches out , pages=

    Rouge: A package for automatic evaluation of summaries , author=. Text summarization branches out , pages=

  47. [49]

    Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments , author=. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages=

  48. [50]

    arXiv preprint arXiv:1904.09675 , year=

    Bertscore: Evaluating text generation with bert , author=. arXiv preprint arXiv:1904.09675 , year=

    Pith/arXiv arXiv 1904

  49. [51]

    Neural Responding Machine for Short-Text Conversation , author=. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

  50. [52]

    Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    A Neural Network Approach to Context-Sensitive Generation of Conversational Responses , author=. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2015

  51. [53]

    Audiocaps: Generating captions for audios in the wild , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    2019

  52. [54]

    MusicLM: Generating Music From Text , author=

  53. [55]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  54. [56]

    AIR -Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

    Yang, Qian and Xu, Jin and Liu, Wenrui and Chu, Yunfei and Jiang, Ziyue and Zhou, Xiaohuan and Leng, Yichong and Lv, Yuanjun and Zhao, Zhou and Zhou, Chang and Zhou, Jingren. AIR -Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

    2024

  55. [57]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=. 2020 , publisher=

    2020

  56. [58]

    ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    The cocktail fork problem: Three-stem audio separation for real-world soundtracks , author=. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2022 , organization=

    2022

  57. [59]

    arXiv preprint arXiv:2406.13340 , year=

    SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words , author=. arXiv preprint arXiv:2406.13340 , year=

    Pith/arXiv arXiv

  58. [60]

    arXiv preprint arXiv:2409.18042 , year=

    Emova: Empowering language models to see, hear and speak with vivid emotions , author=. arXiv preprint arXiv:2409.18042 , year=

    Pith/arXiv arXiv

  59. [61]

    2024 , eprint=

    WavChat: A Survey of Spoken Dialogue Models , author=. 2024 , eprint=

    2024

  60. [62]

    arXiv preprint arXiv:2409.06666 , year=

    Llama-omni: Seamless speech interaction with large language models , author=. arXiv preprint arXiv:2409.06666 , year=

    Pith/arXiv arXiv

  61. [63]

    The Thirteenth International Conference on Learning Representations , year=

    VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words? , author=. The Thirteenth International Conference on Learning Representations , year=

  62. [64]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  63. [65]

    arXiv preprint arXiv:2401.09774 , year=

    On the audio hallucinations in large audio-video language models , author=. arXiv preprint arXiv:2401.09774 , year=

    Pith/arXiv arXiv

  64. [66]

    arXiv preprint arXiv:2503.02318 , year=

    Audio-reasoner: Improving reasoning capability in large audio language models , author=. arXiv preprint arXiv:2503.02318 , year=

    arXiv

  65. [67]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  66. [68]

    doi:10.5281/zenodo.1251982 , url =

    Mark Cartwright and Bongjun Kim and Bryan Pardo , title =. doi:10.5281/zenodo.1251982 , url =

    work page doi:10.5281/zenodo.1251982

  67. [69]

    Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , pages=

    Vocalsketch: Vocally imitating audio concepts , author=. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , pages=

  68. [70]

    arXiv preprint arXiv:2305.10355 , year=

    Evaluating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2305.10355 , year=

    Pith/arXiv arXiv

  69. [71]

    Proceedings of the AAAI Conference on Artificial Intelligence , pages=

    Detecting and preventing hallucinations in large vision language models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , pages=

  70. [72]

    arXiv preprint arXiv:2302.04023 , year=

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity , author=. arXiv preprint arXiv:2302.04023 , year=

    Pith/arXiv arXiv

  71. [73]

    Transactions of the Association for Computational Linguistics , volume=

    Hallucinations in large multilingual translation models , author=. Transactions of the Association for Computational Linguistics , volume=. 2023 , publisher=

    2023

  72. [74]

    arXiv preprint arXiv:2310.08753 , year=

    Compa: Addressing the gap in compositional reasoning in audio-language models , author=. arXiv preprint arXiv:2310.08753 , year=

    Pith/arXiv arXiv

  73. [75]

    ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Can large audio-language models truly hear? Tackling hallucinations with multi-task assessment and stepwise audio reasoning , author=. ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2025 , organization=

    2025

  74. [76]

    2024 Conference of the International Speech Communication Association (INTERSPEECH) , year=

    Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models , author=. 2024 Conference of the International Speech Communication Association (INTERSPEECH) , year=

    2024

  75. [77]

    arXiv preprint arXiv:2412.02612 , year=

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot , author=. arXiv preprint arXiv:2412.02612 , year=

    Pith/arXiv arXiv

  76. [78]

    arXiv preprint arXiv:2310.13289 , year=

    Salmonn: Towards generic hearing abilities for large language models , author=. arXiv preprint arXiv:2310.13289 , year=

    Pith/arXiv arXiv

  77. [79]

    arXiv preprint arXiv:2406.16020 , year=

    Audiobench: A universal benchmark for audio large language models , author=. arXiv preprint arXiv:2406.16020 , year=

    Pith/arXiv arXiv

  78. [80]

    arXiv preprint arXiv:2504.01990 , year=

    Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems , author=. arXiv preprint arXiv:2504.01990 , year=

    Pith/arXiv arXiv

  79. [82]

    arXiv preprint arXiv:2502.11946 , year=

    Step-audio: Unified understanding and generation in intelligent speech interaction , author=. arXiv preprint arXiv:2502.11946 , year=

    Pith/arXiv arXiv

  80. [83]

    arXiv preprint arXiv:2410.18325 , year=

    AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models , author=. arXiv preprint arXiv:2410.18325 , year=

    Pith/arXiv arXiv

Showing first 80 references.