Pith. sign in

REVIEW 16 cited by

Audio-Visual LLM for Video Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.06720 v2 pith:WU5H7LJY submitted 2023-12-11 cs.CV

Fangxun Shu , Lei Zhang , Hao Jiang , Cihang Xie This is my paper · ORCID
classification cs.CV
keywords audio-visualvideoachievestasksunderstandingauditorydatasettraining
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper presents Audio-Visual LLM, a Multimodal Large Language Model that takes both visual and auditory inputs for holistic video understanding. A key design is the modality-augmented training, which involves the integration of modality-specific tokens engineered to activate the appropriate visual and/or auditory encoder selectively. This mechanism is pivotal in enabling end-to-end joint training with video data at different modalities, including visual-only, audio-only, and audio-visual formats. Moreover, we introduce a high-quality video instruction dataset, derived from GPT-4. This dataset allows Audio-Visual LLM to adeptly process a variety of task-oriented video instructions, ranging from multi-turn conversations and audio-visual narratives to complex reasoning tasks. Extensive experiments demonstrate that Audio-Visual LLM impressively achieves strong zero-shot results across a range of video understanding tasks. For example, Audio-Visual LLM achieves an accuracy of 53.7% on MSRVTT-QA, outperforming non-LLM-based InterVideo by 6.6% and LLM-based Valley by 4.4%, respectively. Additionally, our Audio-Visual LLM also achieves competitive performance on audio tasks (e.g., AudioCaps).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Patch-level Sounding Object Tracking for Audio-Visual Question Answering

    cs.MM 2024-12 conditional novelty 7.0 of 10

    A new patch-level sounding object tracking method with motion-, sound-, and question-driven graph modules achieves 78.42% average accuracy on MUSIC-AVQA, competitive with large-scale pretraining approaches.

  2. "Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision People

    cs.HC 2025-06 conditional novelty 6.0 of 10

    Blind and low vision people already use generative AI to protect their visual privacy, and they want future tools to process data locally with zero-retention guarantees and sensitive-content redaction.

  3. RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RAVEN uses query-conditioned token gating plus a new audio-video-sensor QA dataset to improve multimodal question answering, with reported gains of up to 14.5% over prior models.

  4. From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A survey that organizes Omni-MLLMs into four architectural components and a taxonomy of encoding, alignment, and generation methods.

  5. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos

    cs.CV 2024-11 conditional novelty 6.0 of 10

    LongVALE is a new benchmark of 8,411 long videos with 105,730 omni-modal events, each annotated with temporal boundaries and captions that relate visual, audio, and speech, and it shows that a video LLM trained on thi...

  6. VideoOrion: Tokenizing Object Dynamics in Videos

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Encoding video as a small set of object tokens, produced by off-the-shelf detection, segmentation, and tracking models, improves video QA accuracy and enables video-based referring in a 7B video-LLM.

  7. AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

    cs.CV 2025-10 unverdicted novelty 5.0 of 10

    AV-Master reports state-of-the-art accuracy on four audio-visual question answering benchmarks by combining sequential question-guided focus sampling with modality-preference activation.

  8. ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A 7B multimodal model that fuses audio and visual signals with explicit timestamps achieves strong measured comprehension of real-world short videos on the authors' new ShortVid-Bench benchmark.

  9. IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    IntentVCNet uses per-frame object coordinates, red-box visual prompts, and a lightweight box adapter to make video captioning focus on a user-selected object, reporting 225.19 CIDEr on the IntentVC public test set.

  10. Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-CoT contributes a new public dataset and benchmark that add fine-grained chain-of-thought annotations to six spatiotemporal video tasks, with fine-tuning experiments showing moderate gains.

  11. Reinforcing Video Reasoning with Focused Thinking

    cs.CV 2025-05 reject novelty 5.0 of 10

    A GRPO variant with token-level KL weighting and partial-credit rewards improves video-QA on modified multi-answer benchmarks, but transfer to original single-answer benchmarks is not established.

  12. InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

    cs.CV 2024-12 conditional novelty 5.0 of 10

    The authors integrate streaming perception, compressed long-term memory, and a reasoning model into one open-source system, reporting SOTA open-source results on several video and audio benchmarks.

  13. Learning Sparsity for Effective and Efficient Music Performance Question Answering

    cs.SD 2025-06 conditional novelty 4.0 of 10

    Sparsify reports state-of-the-art accuracy on Music AVQA benchmarks by borrowing three existing sparsification techniques, cutting training time by 28% and retaining 70-80% of accuracy on a 25% data subset.

  14. Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing

    cs.CV 2025-05 conditional novelty 4.0 of 10

    Combining larger LLM decoders, preceding-text context, and iterative self-correction reduces character error rate on CNVSRC.Single Chinese visual speech recognition from 49.32% to 38.18%.

  15. Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A literature review cataloging LLM-based augmentation methods across image, text, and speech, with a taxonomy of techniques, limitations, and suggested fixes.

  16. Do Language Models Understand Time?

    cs.CV 2024-12 conditional novelty 3.0 of 10

    A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.

Pith tools