Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Decoupled audio-visual captions force models to listen, not just look, and close the gap on invisible sound events.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:47 UTC pith:W237ARI7

load-bearing objection Solid data recipe for stronger audio grounding in open omni-modal LLMs: tripartite captions heavy on invisible events plus CoT QA and two-stage training give consistent gains on the Qwen2.5-Omni backbone. the 3 major comments →

arxiv 2607.10299 v1 pith:W237ARI7 submitted 2026-07-11 cs.LG

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

Kaiying Yan , Luoyi Sun , Xiao Zhou , Weidi Xie This is my paper
classification cs.LG
keywords omni-modal understandingaudio-visual captioningdecoupled modalitieschain-of-thought reasoninginvisible sound eventsinstruction tuningmultimodal large language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Omni-modal video models remain vision-heavy because training data rarely supply independent, dense auditory supervision; models can often guess sounds from what they see and therefore never learn to hear off-screen or non-speech events. This paper claims that an automated pipeline producing three aligned captions per clip—visual-only, audio-only, and joint—plus a matching set of chain-of-thought questions supplies exactly that missing signal. A two-stage regimen first trains the model to generate the dense captions, then tunes it on the questions, yielding lower miss rates for both visible and invisible sounds and higher accuracy on audio-centric and cross-modal benchmarks without hurting pure visual performance. A sympathetic reader cares because the same data-centric recipe turns an existing open model into one that can localize sirens, blender noise, or off-screen laughter instead of hallucinating from visual priors.

Core claim

The scarcity of datasets that explicitly disentangle and then recombine auditory and visual semantics is the main bottleneck for robust omni-modal understanding. By constructing AVDC (tripartite V/A/AV captions with roughly 37 percent invisible sound events) and AVDC-QA-CoT, then applying caption-generation pre-training followed by instruction tuning, the authors obtain consistent gains on captioning, omni-modal QA, and pure audio QA while preserving visual-only accuracy.

What carries the argument

The AVDC tripartite caption structure (visual-only, audio-only, joint) produced by hierarchical fusion of specialized audio models and large language models, together with the two-stage training curriculum that first aligns encoders via caption generation and then teaches reasoning via CoT QA.

Load-bearing premise

The multi-stage automatic annotation pipeline produces captions and rationales clean enough that residual model biases and errors do not systematically mislead the subsequent fine-tuning.

What would settle it

Train an identical two-stage schedule on a smaller set of the same videos that have been fully re-annotated by humans for every visible and invisible sound event; if the human-annotated version produces no further drop in invisible-sound miss rate on the held-out AVDC-test split, the automatic pipeline is sufficient; a large remaining gap falsifies the claim that the pipeline already supplies reliable supervision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Models trained this way report substantially lower missing rates for both visible and invisible sound events on dense captioning benchmarks.
  • Accuracy rises on omni-modal QA suites that require audio–visual alignment and temporal reasoning, reaching parity with strong proprietary systems on some tasks.
  • Pure audio QA performance also improves, showing the training does not merely teach visual shortcuts.
  • Visual-only video benchmarks stay essentially unchanged, indicating the audio gains do not trade away vision competence.
  • The same data recipe can be reused to add controllable thinking-mode versus direct-answer inference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same tripartite decoupling could be applied to longer, multi-hour videos to test whether off-screen sound density continues to drive gains once temporal context becomes sparse.
  • If invisible-sound supervision is the true lever, simply up-weighting off-screen events inside existing speech-heavy corpora might recover a fraction of the reported lift without a full new pipeline.
  • The residual inter-model disagreement observed before verification suggests future work could treat annotation uncertainty as an explicit training signal rather than filtering it away.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses the visual bias of current multimodal LLMs by releasing AVDC, a 10k-video dataset of long-form (~30 s) videos annotated with tripartite (visual-only, audio-only, joint AV) captions that deliberately include a high fraction (~37 %) of invisible/off-screen sound events, together with AVDC-QA-CoT, a companion set of open-ended and multiple-choice questions equipped with four-step Chain-of-Thought rationales. An automated multi-stage pipeline (specialized audio LLMs + hierarchical LLM fusion + Gemini-2.5-Flash iterative verification) produces the annotations. Starting from Qwen2.5-Omni-7B-Thinker, the authors apply a two-stage curriculum: Stage I freezes the LLM and specializes the audio/visual encoders on global and timestamped caption generation; Stage II performs full-parameter instruction tuning on a balanced mixture of AVDC-QA-CoT, external VQA/AQA data, and controlled audio-visual misalignment examples. Empirical results on AVDC-test, Video-SALMONN-2, UGC-VideoCap, WorldSense, Daily-Omni, Omnibench, ACVUBench, MMAU, MMAR and two vision-only suites show consistent gains over the base model, especially on invisible-sound recall and pure-audio QA, without degrading visual-only accuracy.

Significance. If the reported gains are driven by the new data rather than residual teacher-model artifacts, the work supplies both a concrete resource (dense, explicitly decoupled audio-visual captions with a substantial invisible-event fraction) and a simple, reproducible training recipe that measurably narrows the audio-visual competence gap of open-source omni-modal models. The public release of code and data, the multi-benchmark evaluation suite that includes external public tests, and the ablation isolating Stage-I captions versus Stage-II mixtures constitute useful contributions to the community. The 300-sample human audit of annotation quality is a welcome transparency measure even if its scale remains modest.

major comments (3)
  1. Section 2.1 (Manual evaluation) reports Cohen’s κ = 0.134 between the two primary audio LLMs and per-model F1 ≈ 0.45–0.47 against 300 human-annotated events; after merging and Gemini verification the sound-event F1 rises only to 0.786–0.885. Because Stage I freezes the LLM and specializes the audio/visual encoders precisely on these captions (and Stage II further mixes the derived CoT rationales), residual systematic omissions or teacher-model biases concerning invisible events can be amplified rather than corrected. The paper needs either a larger-scale human audit of the final retained captions (especially invisible-event labels) or an ablation that quantifies how much of the Table 2–4 gains survive when the model is trained on a fully human-verified subset.
  2. Table 6 (ablation) and the main results tables do not isolate the contribution of the ~37 % invisible-event fraction that is repeatedly advertised as the distinctive feature of AVDC. Without a controlled comparison that holds total caption volume fixed while varying the invisible-event density, it remains unclear whether the reported Miss_inv reductions (Table 2) and audio-QA lifts (Table 4) are truly attributable to the decoupled invisible-sound supervision rather than simply to denser overall audio captions or longer training.
  3. Several evaluation metrics themselves rely on GPT-4o / GPT-3.5 automatic scoring (UGC-VideoCap quality scores, Video-SALMONN-2 Miss/Hall rates, AVDC-test event extraction). Given that the same model family (or close relatives) participates in the annotation pipeline, a modest human re-evaluation of a random subset of the automatic judgments would strengthen the claim that the observed gains are not partly circular.
minor comments (5)
  1. Abstract and §1 claim “long-form” understanding, yet the average video length is only 30.1 s (Table 1) with a hard upper bound of 60 s; the terminology should be tempered or the duration distribution clarified.
  2. Figure 1 and the qualitative examples highlight vision-related content in orange and audio-related content in blue, but the color legend is never stated in the caption; a short legend would improve readability.
  3. Table 1 reports #Visible / #Invisible averages obtained by GPT-4o on 1 000 random videos per dataset; the prompt and inter-annotator agreement for this automatic categorization should be supplied in the supplement.
  4. Hyper-parameter table (Table 5) lists peak learning rates and update counts but omits the precise optimizer, weight-decay, and warm-up schedule; these details matter for reproducibility of the two-stage curriculum.
  5. Typographical slips: “Code and dataset are related on …” (abstract), “that that capture” (§2.1), and inconsistent capitalization of model names (Qwen-2.5-Omni vs. Qwen2.5-Omni).

Circularity Check

0 steps flagged

Empirical data-centric paper with no definitional or fitted circularity; only minor shared-LLM usage in automated annotation and some GPT scoring, which does not force the external-benchmark gains.

full rationale

The paper's central claim is that the AVDC tripartite captions (generated via off-the-shelf audio LLMs + hierarchical LLM fusion + Gemini verification) plus AVDC-QA-CoT and two-stage fine-tuning of Qwen2.5-Omni yield measurable gains on external public benchmarks (WorldSense, Daily-Omni, Omnibench, AVUT, MMAU, MMAR, Video-SALMONN-2, UGC-VideoCap, Video-MME, MVBench). These benchmarks pre-exist and are not constructed from AVDC, so reported accuracy/Miss/Hall lifts are not true by construction. There are no equations defining a quantity in terms of itself, no parameters fitted to a subset then re-predicted as a related quantity, no uniqueness theorems imported from the authors, and no ansatz smuggled via self-citation. The only mild loop is the use of related LLM families (Qwen/Gemini/GPT) both to synthesize training captions/CoTs and (in places) to score captions or extract events for AVDC-test; this is standard practice in automated data pipelines and does not make the external-benchmark improvements definitional. Ablations (Table 6) further isolate contributions of Stage I/II and data mixtures without circular reduction. Score 1 reflects only that minor shared-model usage, not load-bearing circularity. Honest non-finding of significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

As an empirical systems paper the load-bearing premises are engineering assumptions about automated annotation quality, the sufficiency of a 7B backbone, and the representativeness of the chosen public benchmarks rather than free mathematical parameters or newly postulated physical entities.

free parameters (5)
  • Stage-I peak learning rate = 1e-5
    Set to 1e-5 by hand; controls encoder adaptation strength.
  • Stage-II peak learning rate = 5e-6
    Set to 5e-6 by hand; controls full-model instruction tuning.
  • segment duration bounds = [2,30] s
    Clips forced into [2,30] s after scene/audio-boundary detection; arbitrary but fixed.
  • dynamic sampling interval = 0.5/1 s
    0.5 s for videos <30 s, 1 s otherwise; chosen for compute/resolution trade-off.
  • quality-score retention threshold = ≥4/5
    Only captions scoring ≥4/5 after Gemini verification are kept.
axioms (4)
  • domain assumption Off-the-shelf audio LLMs (Qwen2-Audio, Audio-Flamingo 2, SALMONN, MU-LLaMA) plus LLM fusion produce sufficiently accurate raw event descriptions for downstream supervision.
    Invoked throughout §2.1; the paper’s own Cohen’s κ=0.134 shows the assumption is only partially true before verification.
  • domain assumption Gemini-2.5-Flash iterative verification-correction reliably removes hallucinations and omissions while preserving correct content.
    Core of the quality-control loop in §2.1; final F1 numbers rest on this step.
  • ad hoc to paper DeepSeek-generated CoT rationales that follow the four-step template (decomposition, temporal grounding, visual/audio perception, multimodal synthesis) constitute faithful training signals for reasoning.
    Used to construct AVDC-QA-CoT (§2.2); no external validation of rationale faithfulness is supplied.
  • domain assumption Filtering videos to those containing ≥3 environmental sound events totaling >2 s and >10 % of duration yields a representative long-form omni-modal distribution.
    Pre-processing step in §2.1; selection criterion is stated but not shown to be free of domain bias.
invented entities (2)
  • AVDC tripartite caption schema (V / A / AV with timestamps) independent evidence
    purpose: Force models to learn modality-specific and cross-modal representations, especially for invisible sounds.
    New data format introduced by the paper; independent evidence is the public release claim and the measured reduction in Miss_inv.
  • AVDC-QA-CoT structured four-step CoT rationales no independent evidence
    purpose: Supply explicit multi-step audio-visual reasoning supervision.
    Synthetic construct generated by prompting DeepSeek; no external corpus of human CoTs of this form is cited.

pith-pipeline@v1.1.0-grok45 · 28590 in / 3414 out tokens · 32330 ms · 2026-07-14T12:47:55.586550+00:00 · methodology

0 comments
read the original abstract

Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues. To bridgethis gap, we present AVDC (Audio-Visual Decoupled Captions), a large-scaledataset designed to disentangle visual and auditory semantics. Specifi-cally, we propose an automated pipeline that leverages off-the-shelf mod-els to annotate videos with tripartite captions: visual-only (V), audio-only (A), and joint audio-visual (AV). This decoupled structure explic-itly captures both modality-specific nuances and complex cross-modalinteractions. Building upon this, we introduce AVDC-QA-CoT, a Chain-of-Thought augmented question-answering dataset to foster audio-visualreasoning. To fully exploit these resources, we employ a two-stage train-ing paradigm: omni-modal caption generation pre-training on AVDC, fol-lowed by instruction tuning on AVDC-QA-CoT. Extensive experiments acrossdiverse downstream tasks, spanning video captioning, audio-centric anal-ysis, and omni-modal benchmarks, demonstrate consistent and signifi-cant performance gains, showing the efficacy of our proposed datasetsand training strategy in advancing omni-modal perception. Code anddataset are related on https://radiant0726.github.io/AVDC-web/.

Figures

Figures reproduced from arXiv: 2607.10299 by Kaiying Yan, Luoyi Sun, Weidi Xie, Xiao Zhou.

Figure 1
Figure 1. Figure 1: Dataset and qualitative results. (a) AVDC dataset. An audio-visual cap￾tion dataset with balanced information across modalities, (b) AVDC-QA-CoT dataset. It include QA pairs with chain-of-thought reasoning. (c) QA results visualization. The comparison demonstrates that our model generates more detailed and accurate cap￾tions with fewer missing or hallucinated events, and exhibits more coherent omni-modal r… view at source ↗
Figure 2
Figure 2. Figure 2: Automatic pipeline for the audio-visual decoupled caption gener [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model architecture. The framework preserves fine-grained audio-visual cor￾respondence via feature preprocessing and employs a two-stage training strategy: Stage I. Omni-modal caption generation training to optimal multimodal representations, and Stage II. Instruction Tuning for QA, supporting both direct answers and chain-of￾thought reasoning. Abbreviations: Cap. (Caption), Ans. (Answer). Preprocessing str… view at source ↗
Figure 4
Figure 4. Figure 4: We use ATST-F [42] to detect all sound events in videos, leveraging the [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of Caption Lengths and Video Durations richness and temporal coverage, with captions that are semantically informative and videos long enough to capture meaningful event dynamics. 7.2 Experimental Details In this section, we present the hyperparameters for the two-stage training and the composition of the instruction-tuning data. Training Details Our training strategy adopts a two-stage paradi… view at source ↗
Figure 7
Figure 7. Figure 7: Data statistics of the instruction-tuning data. It covers a broad range of question types, including those with or without chain-of-thought reasoning, as well as multiple￾choice and open-ended questions. Abbreviations: AVQA(audio-visual QA), VQA(video QA), AQA(audio QA), MC (multiple-choice), OE(open-ended). include chain-of-thought rationales. Overall, we provide a diverse, balanced, and reasoning-rich co… view at source ↗
Figure 8
Figure 8. Figure 8: Example of a multiple-choice problem. Time-related content is shown in red, and visual and audio cues in orange and blue [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example of open-ended problem. Time-related content is shown in red, and visual and audio cues in orange and blue [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example of QA results. The parts highlighted in <red> and [green] indicate thinking-answer process and reasoning steps [PITH_FULL_IMAGE:figures/full_fig_p030_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Example of QA results. The parts highlighted in <red> and [green] indicate thinking-answer process and reasoning steps [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 25 linked inside Pith

  1. [1]

    In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization

    Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with im- proved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summa- rization. pp. 65–72 (2005)

    2005

  2. [2]

    arXiv preprint arXiv:2402.12451 (2024)

    Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Cornia, M., Cucchiara, R.: The revolution of multimodal large language models: a survey. arXiv preprint arXiv:2402.12451 (2024)

    arXiv 2024

  3. [3]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chen, J., Zeng, Z., Lin, Y., Li, W., Ma, Z., Shou, M.Z.: Livecc: Learning video llm with streaming speech transcription at scale. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29083–29095 (2025)

    2025

  4. [4]

    Advances in Neural Information Processing Systems37, 19472–19495 (2024)

    Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Tang, Z., Yuan, L., et al.: Sharegpt4video: Improving video understanding and genera- tion with better captions. Advances in Neural Information Processing Systems37, 19472–19495 (2024)

    2024

  5. [5]

    arXiv preprint arXiv:2304.08345 (2023)

    Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., Liu, J.: Valor: Vision- audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 (2023)

    Pith/arXiv arXiv 2023

  6. [6]

    Advances in Neural Information Processing Systems36, 72842–72866 (2023)

    Chen,S.,Li,H.,Wang,Q.,Zhao,Z.,Sun,M.,Zhu,X.,Liu,J.:Vast:Avision-audio- subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems36, 72842–72866 (2023)

    2023

  7. [7]

    Cheng, X., Zhang, Z., Wang, Z., Fang, M., Huang, R., Zheng, S., Hu, R., Jionghao, B., Jin, T., Zhao, Z.: Avset-10m: An open large-scale audio-visual dataset with high correspondence (2024)

    2024

  8. [8]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    arXiv preprint arXiv:2407.20962 (2024)

    Chi, X., Wang, Y., Cheng, A., Fang, P., Tian, Z., He, Y., Liu, Z., Qi, X., Pan, J., Zhang, R., et al.: Mmtrail: A multimodal trailer video dataset with language and music descriptions. arXiv preprint arXiv:2407.20962 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    arXiv preprint arXiv:2407.10759 (2024)

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al.: Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    arXiv preprint arXiv:2504.18425 (2025)

    Ding, D., Ju, Z., Leng, Y., Liu, S., Liu, T., Shang, Z., Shen, K., Song, W., Tan, X., Tang, H., et al.: Kimi-audio technical report. arXiv preprint arXiv:2504.18425 (2025)

    Pith/arXiv arXiv 2025

  13. [13]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Du, H., Li, G., Zhou, C., Zhang, C., Zhao, A., Hu, D.: Crab: A unified audio- visual scene understanding model with explicit cooperation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18804–18814 (2025)

    2025

  14. [14]

    arXiv preprint arXiv:2503.21776 (2025)

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24108–24118 (2025) 16

    2025

  16. [16]

    In: Proceedings of the IEEE international conference on computer vision

    Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: Temporal activity localization via language query. In: Proceedings of the IEEE international conference on computer vision. pp. 5267–5275 (2017)

    2017

  17. [17]

    In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)

    Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 776–780. IEEE (2017)

    2017

  18. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Geng, T., Wang, T., Duan, J., Cong, R., Zheng, F.: Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22942–22951 (2023)

    2023

  19. [19]

    In: Proceedings of the Computer Vision and Pattern Recognition Con- ference

    Geng, T., Zhang, J., Wang, Q., Wang, T., Duan, J., Zheng, F.: Longvale: Vision- audio-language-event benchmark towards time-aware omni-modal perception of long videos. In: Proceedings of the Computer Vision and Pattern Recognition Con- ference. pp. 18959–18969 (2025)

    2025

  20. [20]

    arXiv preprint arXiv:2503.03983 (2025)

    Ghosh, S., Kong, Z., Kumar, S., Sakshi, S., Kim, J., Ping, W., Valle, R., Manocha, D., Catanzaro, B.: Audio flamingo 2: An audio-language model with long-audio un- derstanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    arXiv preprint arXiv:2406.11768 (2024)

    Ghosh, S., Kumar, S., Seth, A., Evuru, C.K.R., Tyagi, U., Sakshi, S., Nieto, O., Duraiswami, R., Manocha, D.: Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406.11768 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    arXiv preprint arXiv:2507.08128 (2025)

    Goel, A., Ghosh, S., Kim, J., Kumar, S., Kong, Z., Lee, S.g., Yang, C.H.H., Duraiswami, R., Manocha, D., Valle, R., et al.: Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    arXiv preprint arXiv:2502.04326 (2025)

    Hong, J., Yan, S., Cai, J., Jiang, X., Hu, Y., Xie, W.: Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326 (2025)

    Pith/arXiv arXiv 2025

  24. [24]

    arXiv preprint arXiv:2503.24164 (2025)

    Huynh, N.D., Bouadjenek, M.R., Razzak, I., Hacid, H., Aryal, S.: Svla: A unified speech-vision-language assistant with multimodal reasoning and speech generation. arXiv preprint arXiv:2503.24164 (2025)

    Pith/arXiv arXiv 2025

  25. [25]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Lee, S., Chung, J., Yu, Y., Kim, G., Breuel, T., Chechik, G., Song, Y.: Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10274–10284 (2021)

    2021

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.R., Hu, D.: Learning to answer questions in dynamic audio-visual scenarios. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19108–19118 (2022)

    2022

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024)

    2024

  28. [28]

    In: 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

    Li, X., Takamichi, S., Saeki, T., Chen, W., Shiota, S., Watanabe, S.: Yodas: Youtube-oriented dataset for audio and speech. In: 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 1–8. IEEE (2023)

    2023

  29. [29]

    arXiv preprint arXiv:2501.15368 (2025) 17

    Li, Y., Liu, J., Zhang, T., Chen, S., Li, T., Li, Z., Liu, L., Ming, L., Dong, G., Pan, D., et al.: Baichuan-omni-1.5 technical report. arXiv preprint arXiv:2501.15368 (2025) 17

    arXiv 2025

  30. [30]

    arXiv preprint arXiv:2409.15272 (2024)

    Li, Y., Zhang, G., Ma, Y., Yuan, R., Zhu, K., Guo, H., Liang, Y., Liu, J., Wang, Z., Yang, J., et al.: Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272 (2024)

    arXiv 2024

  31. [31]

    In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP)

    Liu, S., Hussain, A.S., Sun, C., Shan, Y.: Music understanding llama: Advanc- ing text-to-music generation with question answering and captioning. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). pp. 286–290. IEEE (2024)

    2024

  32. [32]

    arXiv e-prints pp

    Liu, Z., Dong, Y., Wang, J., Liu, Z., Hu, W., Lu, J., Rao, Y.: Ola: Pushing the fron- tiers of omni-modal language model with progressive modality alignment. arXiv e-prints pp. arXiv–2502 (2025)

    2025

  33. [33]

    arXiv preprint arXiv:2506.05328 (2025)

    Lu, L., Chen, G., Li, Z., Liu, Y., Lu, T.: Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for mllms. arXiv preprint arXiv:2506.05328 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    arXiv preprint arXiv:2505.13032 (2025)

    Ma, Z., Ma, Y., Zhu, Y., Yang, C., Chao, Y.W., Xu, R., Chen, W., Chen, Y., Chen, Z., Cong, J., et al.: Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032 (2025)

    Pith/arXiv arXiv 2025

  35. [35]

    arXiv preprint arXiv:2510.12720 (2025)

    Ma, Z., Xu, R., Xing, Z., Chu, Y., Wang, Y., He, J., Xu, J., Heng, P.A., Yu, K., Lin, J., et al.: Omni-captioner: Data pipeline, models, and benchmark for omni detailed perception. arXiv preprint arXiv:2510.12720 (2025)

    arXiv 2025

  36. [36]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024)

    2024

  37. [37]

    arXiv preprint arXiv:2405.03770 (2024)

    Madan, N., Møgelmose, A., Modi, R., Rawat, Y.S., Moeslund, T.B.: Foundation models for video understanding: A survey. arXiv preprint arXiv:2405.03770 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    ArXiv 2306.05424 (2023)

    Muhammad Maaz, Hanoona Rasheed, S.K., Khan, F.: Video-chatgpt: Towards de- tailed video understanding via large vision and language models. ArXiv 2306.05424 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    In: Findings of the Association for Compu- tational Linguistics: ACL 2024

    Nguyen, T., Bin, Y., Xiao, J., Qu, L., Li, Y., Wu, J.Z., Nguyen, C.D., Ng, S.K., Tuan, L.A.: Video-language understanding: A survey from model architecture, model training, and data perspectives. In: Findings of the Association for Compu- tational Linguistics: ACL 2024. pp. 3636–3657 (2024)

    2024

  40. [40]

    ORCHESTRATION, D.B.T.: Avocado: An audiovisual video captioner driven by temporal orchestration

  41. [41]

    arXiv preprint arXiv:2410.19168 (2024)

    Sakshi, S., Tyagi, U., Kumar, S., Seth, A., Selvakumar, R., Nieto, O., Duraiswami, R., Ghosh, S., Manocha, D.: Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP)

    Schmid, F., Morocutti, T., Foscarin, F., Schlüter, J., Primus, P., Widmer, G.: Ef- fective pre-training of audio transformers for sound event detection. In: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). pp. 1–5. IEEE (2025)

    2025

  43. [43]

    IEEE Open Journal of Signal Pro- cessing (2025)

    Sudarsanam, P., Martín-Morató, I., Hakala, A., Virtanen, T.: Avcaps: An audio- visual dataset with modality-specific captions. IEEE Open Journal of Signal Pro- cessing (2025)

    2025

  44. [44]

    arXiv preprint arXiv:2506.15220 (2025)

    Tang, C., Li, Y., Yang, Y., Zhuang, J., Sun, G., Li, W., Ma, Z., Zhang, C.: video-salmonn 2: Captioning-enhanced audio-visual large language models. arXiv preprint arXiv:2506.15220 (2025)

    arXiv 2025

  45. [45]

    arXiv preprint arXiv:2310.13289 (2023) 18

    Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., Zhang, C.: Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289 (2023) 18

    Pith/arXiv arXiv 2023

  46. [46]

    arXiv preprint arXiv:2312.11805 (2023)

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  47. [47]

    Team, O.M.o.: Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multi- modal live streaming on your phone (2025)

    2025

  48. [48]

    arXiv preprint arXiv:2505.14142 (2025)

    Wijngaard, G., Formisano, E., Esposito, M., Dumontier, M.: Audsemthinker: En- hancing audio-language models through reasoning over semantics of sound. arXiv preprint arXiv:2505.14142 (2025)

    arXiv 2025

  49. [49]

    arXiv preprint arXiv:2507.11336 (2025)

    Wu, P., Liu, Y., Zhu, Z., Zhou, E., Shen, J.: Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks. arXiv preprint arXiv:2507.11336 (2025)

    arXiv 2025

  50. [50]

    In: Proc

    Wu, Y., Lu, Y., Chen, Y., Song, J., Chen, W., Song, R., Watanabe, S.: Galaxy: A large-scale open-domain dataset for multimodal learning. In: Proc. Interspeech

  51. [51]

    5-omni technical report

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al.: Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    arXiv preprint arXiv:2509.17765 (2025)

    Xu, J., Guo, Z., Hu, H., Chu, Y., Wang, X., He, J., Wang, Y., Shi, X., He, T., Zhu, X., et al.: Qwen3-omni technical report. arXiv preprint arXiv:2509.17765 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    arXiv preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  54. [54]

    Advances in Neural Information Processing Systems37, 57240–57261 (2024)

    Yang, D., Huang, S., Lu, C., Han, X., Zhang, H., Gao, Y., Hu, Y., Zhao, H.: Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems37, 57240–57261 (2024)

    2024

  55. [55]

    In: Proceedings of the 30th ACM international conference on multimedia

    Yang, P., Wang, X., Duan, X., Chen, H., Hou, R., Jin, C., Zhu, W.: Avqa: A dataset for audio-visual question answering on videos. In: Proceedings of the 30th ACM international conference on multimedia. pp. 3480–3491 (2022)

    2022

  56. [56]

    arXiv preprint arXiv:2506.21277 (2025)

    Yang, Q., Yao, S., Chen, W., Fu, S., Bai, D., Zhao, J., Sun, B., Yin, B., Wei, X., Zhou, J.: Humanomniv2: From understanding to omni-modal reasoning with context. arXiv preprint arXiv:2506.21277 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    arXiv preprint arXiv:2503.19951 (2025)

    Yang, Y., Zhuang, J., Sun, G., Tang, C., Li, Y., Li, P., Jiang, Y., Li, W., Ma, Z., Zhang, C.: Acvubench: Audio-centric video understanding benchmark. arXiv preprint arXiv:2503.19951 (2025)

    arXiv 2025

  58. [58]

    arXiv preprint arXiv:2306.02858 (2023)

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

    Pith/arXiv arXiv 2023

  59. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Zhou,Z.,Mei,K.,Lu,Y.,Wang,T.,Rao,F.:Harmonyset:Acomprehensivedataset for understanding video-music semantic alignment and temporal synchronization. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 3152–3162 (2025)

    2025

  60. [60]

    arXiv preprint arXiv:2505.17862 (2025)

    Zhou, Z., Wang, R., Wu, Z.: Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862 (2025)

    arXiv 2025

  61. [61]

    Zou, H., Luo, T., Xie, G., Lv, F., Wang, G., Chen, J., Wang, Z., Zhang, H., Zhang, H., et al.: From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding. arXiv preprint arXiv:2409.18938 (2024) 1 7 Supplementary 7.1 AVDC Dataset Statistics In this section, we conduct a systematic analysis of the proposed AV...

    Pith/arXiv arXiv 2024

  62. [62]

    **Error list**: for each error, provide the problem type, and a brief 8 explanation

  63. [63]

    1/5, 2/5, ... 5/5

    **Overall quality score**: a number from 1 to 5 (use “1/5, 2/5, ... 5/5”) that reflects the overall accuracy, completeness and richness of details in both audio and visual aspects of the video

  64. [64]

    errors": [ {

    **Rewritten caption**: Produce a corrected version of the caption, **only modifying** the parts that are wrong, remove meaningless back- ground noise descriptions, and reuse all the originally correct sentences. output format: { "errors": [ { "type": "[Problem Type]", "explanation": "[Explanation]" }], "score": [Overall quality score], "rewritten caption"...

  65. [65]

    Question Decomposition: Break the question into smaller, manage- able sub-questions without explicitly referencing or revealing elements that appear in the correct answer

  66. [66]

    Track how events unfold over time and highlight when key actions or changes occur

    Temporal Grounding: Identify the relevant moments (timestamps or segments) in the video for each sub-question. Track how events unfold over time and highlight when key actions or changes occur

  67. [67]

    Visual Perception: At the grounded moments, analyze visual through steps such as object detection, scene understanding, and action recognition

  68. [68]

    Audio Perception: At the grounded moments, analyze audio through steps such as sound event detection, speech understanding, and music analysis

  69. [69]

    question

    Multimodal Reasoning and Answer Synthesis: Integrate visual and audio cues at the relevant timepoints to cross-validate observations, understand context, and derive the final answer clearly. Output format (JSON): { "question": "<your question>", "answer": "<your answer>", "reasoning": { "question decomposition": "...", "temporal grounding": "...", "visual...