REVIEW 6 cited by
VideoLLM: Modeling Video Sequence with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the exponential growth of video data, there is an urgent need for automated technology to analyze and comprehend video content. However, existing video understanding models are often task-specific and lack a comprehensive capability of handling diverse tasks. The success of large language models (LLMs) like GPT has demonstrated their impressive abilities in sequence causal reasoning. Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding. VideoLLM incorporates a carefully designed Modality Encoder and Semantic Translator, which convert inputs from various modalities into a unified token sequence. This token sequence is then fed into a decoder-only LLM. Subsequently, with the aid of a simple task head, our VideoLLM yields an effective unified framework for different kinds of video understanding tasks. To evaluate the efficacy of VideoLLM, we conduct extensive experiments using multiple LLMs and fine-tuning methods. We evaluate our VideoLLM on eight tasks sourced from four different datasets. The experimental results demonstrate that the understanding and reasoning capabilities of LLMs can be effectively transferred to video understanding tasks. We release the code at https://github.com/cg1177/VideoLLM.
Forward citations
Cited by 6 Pith papers
-
GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs
GMoT's gated motion tokens improve multimodal LLM micro-gesture recognition on iMiGUE and SMG, with limited support for reasoning-grounding claims.
-
Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Models
Adding a backward prediction task to LLM training improves long-term action anticipation on Ego4D, lowering edit distance for predicted action sequences.
-
IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
IntentVCNet uses per-frame object coordinates, red-box visual prompts, and a lightweight box adapter to make video captioning focus on a user-selected object, reporting 225.19 CIDEr on the IntentVC public test set.
-
Time-Scaling State-Space Models for Dense Video Captioning
A state-space model with transfer state processes videos chunk by chunk, carries the hidden state forward, and performs online dense video captioning with the same state as full-sequence processing.
-
NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding
NeuroVoxel-LM combines dynamic multi-resolution voxelization with attention-based pooling of NeRF weights, reporting faster 3D feature extraction and modestly better NeRF captioning than fixed-resolution and max-pooli...
-
Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision
A comprehensive review of cross-view video understanding that uses both first-person and third-person cameras, organized into a three-direction taxonomy with a dataset catalog and future research gaps.
Discussion (0). Sign in to comment.