REVIEW 7 cited by
Adaptive Keyframe Sampling for Long Video Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal large language models (MLLMs) have enabled open-world visual understanding by injecting visual input as extra tokens into large language models (LLMs) as contexts. However, when the visual input changes from a single image to a long video, the above paradigm encounters difficulty because the vast amount of video tokens has significantly exceeded the maximal capacity of MLLMs. Therefore, existing video-based MLLMs are mostly established upon sampling a small portion of tokens from input data, which can cause key information to be lost and thus produce incorrect answers. This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS). It inserts a plug-and-play module known as keyframe selection, which aims to maximize the useful information with a fixed number of video tokens. We formulate keyframe selection as an optimization involving (1) the relevance between the keyframes and the prompt, and (2) the coverage of the keyframes over the video, and present an adaptive algorithm to approximate the best solution. Experiments on two long video understanding benchmarks validate that Adaptive Keyframe Sampling improves video QA accuracy (beyond strong baselines) upon selecting informative keyframes. Our study reveals the importance of information pre-filtering in video-based MLLMs. Code is available at https://github.com/ncTimTang/AKS.
Forward citations
Cited by 7 Pith papers
-
MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning
On MarineEVT, an event-centric 20K-pair marine video QA benchmark, EVT-R1 with tool-integrated RL scores 48.89 average accuracy, 5.22 points above the best untuned open-source VLM and 8.54 above the best tool-using co...
-
DynTrace: Tracking Dynamic Object Evidence for 4D Spatio-Temporal Reasoning in MLLMs
A training-free pipeline that feeds MLLMs reprojected motion arrows plus a structured trace graph lifts 4D spatio-temporal QA accuracy on three benchmarks.
-
Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs
MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.
-
Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning
Question-guided dual geometric memories with relevance-novelty utility reportedly reach state-of-the-art video spatial reasoning on two in-domain and five out-of-distribution benchmarks.
-
DATE: Dynamic Absolute Time Enhancement for Long Video Understanding
DATE combines inference-time timestamp token injection with a caption-rewritten, temporally regularized CLIP sampling strategy to improve absolute time reasoning and event localization in long videos.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning
ViaRL uses rule-based reinforcement learning to train a frame selector for video QA, improving Qwen2.5-VL on VideoMME, LVBench, and MLVU by several points.
Discussion (0). Continue with ORCID to comment.