REVIEW 4 cited by
LinVT: Empower Your Image-level Large Language Model to Understand Videos
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have been widely used in various tasks, motivating us to develop an LLM-based assistant for videos. Instead of training from scratch, we propose a module to transform arbitrary well-trained image-based LLMs into video-LLMs (after being trained on video data). To better adapt image-LLMs for processing videos, we introduce two design principles: linear transformation to preserve the original visual-language alignment and representative information condensation from redundant video content. Guided by these principles, we propose a plug-and-play Linear Video Tokenizer(LinVT), which enables existing image-LLMs to understand videos. We benchmark LinVT with six recent visual LLMs: Aquila, Blip-3, InternVL2, Mipha, Molmo and Qwen2-VL, showcasing the high compatibility of LinVT. LinVT-based LLMs achieve state-of-the-art performance across various video benchmarks, illustrating the effectiveness of LinVT in multi-modal video understanding.
Forward citations
Cited by 4 Pith papers
-
DisTime: Distribution-based Time Representation for Video Large Language Models
A single learnable time token, decoded into a probability distribution over time bins, improves temporal grounding in Video-LLMs and is trained partly on a new 1.25M-event pseudo-labeled dataset.
-
AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos
AROMA pairs a blind cook's spoken descriptions of what they feel, smell, and taste with a wearable camera and a video recipe to answer questions and issue proactive alerts, and eight participants rated it usable despi...
-
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.
-
${\mu}^2$Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation
A tokenizer that combines multi-scale CT image features with text questions, plus DPO training on a clinical metric, is claimed to improve automated radiology report generation.
Discussion (0). Sign in to comment.