REVIEW 8 cited by
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME.
Forward citations
Cited by 8 Pith papers
-
ARGUS: Hallucination and Omission Evaluation in Video-LLMs
ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.
-
Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs
MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.
-
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.
-
"Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth
The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.
-
ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments
ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.
-
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Video MLLMs mostly ignore motion in pixel-level visual grounding; a new motion-centric benchmark shows large performance drops.
Discussion (0). Sign in to comment.