REVIEW 31 cited by
Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Multi-modal large language models (MLLMs) have demonstrated considerable potential across various downstream tasks that require cross-domain knowledge. MLLMs capable of processing videos, known as Video-MLLMs, have attracted broad interest in video-language understanding. However, videos, especially long videos, contain more visual tokens than images, making them difficult for LLMs to process. Existing works either downsample visual features or extend the LLM context size, risking the loss of high-resolution information or slowing down inference speed. To address these limitations, we apply cross-attention layers in the intermediate projector between the visual encoder and the large language model (LLM). As the naive cross-attention mechanism is insensitive to temporal order, we further introduce causal cross-attention masks (CCAMs) within the cross-attention layers. This Video-MLLM, named Video-CCAM, is trained in a straightforward two-stage fashion: feature alignment and visual instruction tuning. We develop several Video-CCAM models based on LLMs of different sizes (4B, 9B, and 14B). Video-CCAM proves to be a robust Video-MLLM and shows outstanding performance from short videos to long ones. Among standard video benchmarks like MVBench and VideoChatGPT-QA, Video-CCAM shows outstanding performances (1st/2nd/3rd in MVBench and TGIF-QA, 2nd/3rd/4th in MSVD-QA, MSRVTT-QA, and ActivityNet-QA). In benchmarks encompassing long videos, Video-CCAM models can be directly adapted to long video understanding and still achieve exceptional scores despite being trained solely with images and 16-frame videos. Using 96 frames (6$\times$ the training number of frames), Video-CCAM models rank 1st/2nd/3rd in VideoVista and 1st/2nd/4th in MLVU among all open-source Video-MLLMs, respectively. The code is publicly available in \url{https://github.com/QQ-MM/Video-CCAM}.
Forward citations
Cited by 31 Pith papers
-
Online Video Understanding: OVBench and VideoChat-Online
A new benchmark and a memory-bank architecture let a 4B video AI outperform larger offline and streaming models on online video understanding tasks.
-
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
EviSelect uses the target multimodal model's internal attention as a prior to dynamically select frames, sampling rates, and resolutions, achieving about 50% token reduction and a 3.9x speedup with better benchmark accuracy.
-
Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
Attention maps from a small MLLM can serve as a training-free, query-conditioned frame selector, improving long-video QA accuracy under fixed frame budgets.
-
TimeThink: Reasoning with Time for Video LLMs
TimeThink adds step-wise temporal process rewards (max IoU of referenced intervals) to GRPO for Video-LLMs, improving grounding and reasoning over outcome-only RL baselines.
-
Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
ReMem improves zero-shot long-video QA by combining LLM-based temporal granularity parsing, CLIP-based dual-semantic frame scoring, and structure-aware dynamic frame routing.
-
Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
A multi-turn reinforcement learning framework where a 7B video LLM iteratively retrieves relevant video segments, guided by a gated bi-level reward, achieves state-of-the-art accuracy on VideoMME, MLVU, and EgoSchema ...
-
VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering
A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.
-
LongAnimation: Long Animation Generation with Dynamic Global-Local Memory
LongAnimation uses a dynamic global-local memory, built from a long-video-understanding model's KV cache, to colorize animation sequences of about 500 frames with stable color consistency.
-
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.
-
ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning
ViaRL uses rule-based reinforcement learning to train a frame selector for video QA, improving Qwen2.5-VL on VideoMME, LVBench, and MLVU by several points.
-
Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Data
A single transformer with separate pixel and time vocabularies generates realistic Cherenkov detector hits and supports particle identification and noise filtering after fine-tuning.
-
TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos
A streaming video assistant drops temporally redundant tokens (over 80%) with minimal accuracy loss and uses the drop-ratio curve to trigger proactive responses at scene changes.
-
VEU-Bench: Towards Comprehensive Understanding of Video Editing
A new 19-task video editing benchmark shows that current video LLMs struggle to understand editing concepts, and a model fine-tuned on the benchmark improves both editing and general video reasoning.
-
Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis
FaceCLIP-SDXL encodes identity and text into a single joint embedding and fully fine-tunes Stable Diffusion XL on it, reporting higher face similarity and text alignment than InstantID and PuLID-SDXL.
-
Temporal Preference Optimization for Long-Form Video Understanding
TPO trains video-LMMs to prefer answers generated from complete, relevant frames over answers from incomplete or irrelevant frames, improving temporal grounding on LongVideoBench, MLVU, and Video-MME.
-
CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
A 12,129-question benchmark for long videos that requires models to retrieve the specific video moments supporting each answer, exposing a gap between multiple-choice accuracy and genuine video understanding.
-
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.
-
PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
PVC unifies image and video token compression in VLMs by repeating images as static videos and using causal temporal attention with adaptive compression, achieving strong benchmark results at 64 tokens per frame.
-
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation
VISTA creates synthetic long and high-resolution video instruction data by combining existing clips, and finetuning video LMMs on it improves their accuracy on long-video and high-resolution benchmarks.
-
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
An automated arena benchmark that simulates real users asking open-ended video questions, uses GPT-4o as judge, and ranks 11 large multimodal models via ELO ratings.
-
Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
A CLIP-scored, Gumbel-Max frame sampler with per-frame multi-resolution allocation improves long-video question answering in Video-LLMs under a fixed token budget.
-
Task-Aware KV Compression For Cost-Effective Long Video Understanding
Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.
-
Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.
-
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.
-
VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
An open-ended short-answer long-video benchmark, built by converting MCQ questions from four existing tests, shows large accuracy drops and different model rankings versus multiple-choice evaluation.
-
CoS: Chain-of-Shot Prompting for Long Video Understanding
A training-free method that uses an AI model's yes/no judgments on mosaic clips to build positive and negative shot sets, improving long-video question answering by a few points.
-
Ola: Pushing the Frontiers of Omni-Modal Language Model
Ola, a 7B omni-modal language model, achieves competitive image, video, and audio understanding with progressive modality alignment, though it does not beat all specialized models on every benchmark.
-
Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
A video LLM that adaptively compresses each frame's visual tokens, trained on 2M synthetic GPT-generated QA pairs, outperforms LLaVA-OneVision on VideoMME and MuirBench.
-
LinVT: Empower Your Image-level Large Language Model to Understand Videos
A plug-and-play linear video tokenizer converts existing image-based LLMs into video-understanding LLMs by condensing frames into weighted-average tokens while preserving image capabilities.
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs
LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.
-
Do Language Models Understand Time?
A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.
Discussion (0). Continue with ORCID to comment.