Pith. sign in

REVIEW 2 major objections 1 minor 110 cited by

MLVU: Benchmarking Multi-task Long Video Understanding

T0 review · 2 major / 1 minor · reviewed 2026-05-14 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MLVU benchmark shows current multimodal models struggle with most long video tasks and degrade sharply on longer clips.

desk verdict MLVU adds flexible long durations and genre breadth to video benchmarks but the length-degradation claim rests on unstated controls for frames and tokens. read the letter →

arxiv 2406.04264 v3 pith:T7EF4SUO submitted 2024-06-06 cs.CV cs.AIcs.CL

Junjie Zhou , Yan Shu , Bo Zhao , Boya Wu , Zhengyang Liang , Shitao Xiao , Minghao Qin , Xi Yang
show 4 more authors
Yongping Xiong Bo Zhang Tiejun Huang Zheng Liu
This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords longvideounderstandingmultimodallargelanguagemodelsbenchmarkevaluationanalysiscontextlengthmodelperformancediversegenres
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MLVU as a benchmark to evaluate long video understanding in multimodal large language models. It extends video durations across a wide range, incorporates many genres such as movies and surveillance footage, and defines multiple tasks that test core abilities like summarization and question answering. Testing 23 recent models reveals that all of them fail on most tasks, with clear drops in accuracy as video length increases. A sympathetic reader would care because real applications often require processing extended video where today's techniques appear limited, pointing to specific factors like context handling that need work.

What carries the argument

The MLVU benchmark itself, which supplies extended videos, genre variety, and diversified tasks to measure MLLMs' long-video understanding abilities.

What would settle it

A new model that maintains high accuracy on all MLVU tasks without measurable drop when video length increases would directly challenge the reported performance degradation.

Watch

Extended reading notes

Core claim

MLVU addresses prior benchmark limits by allowing flexible video lengths, covering diverse genres, and offering varied evaluation tasks; the study of 23 MLLMs shows every existing method struggles on most tasks and suffers severe performance degradation on longer videos, while also indicating that context length, image-understanding ability, and LLM backbone choice matter for progress.

Load-bearing premise

The chosen video lengths, genres, and tasks sufficiently represent the main challenges of real-world long video understanding.

Editorial extensions

If this is right

  • Improvements in context length handling would directly raise scores on longer videos.
  • Stronger image-understanding components and better LLM backbones would lift overall long-video performance.
  • Models must be tested across multiple genres to confirm they generalize beyond narrow cases.
  • Future work can use MLVU scores to track whether new techniques close the observed gaps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Success on MLVU could transfer to practical uses such as analyzing surveillance or summarizing extended recordings.
  • The benchmark may highlight whether scaling context windows alone solves the length degradation or if new architectures are required.
  • Adding even longer videos or additional task types in follow-ups would likely expose further limits in current designs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces the MLVU benchmark for multi-task long video understanding, extending video lengths substantially, incorporating diverse genres (movies, surveillance, egocentric, cartoons, games), and defining multiple evaluation tasks to probe MLLM capabilities. It reports results from 23 recent MLLMs showing that all models struggle on most tasks and exhibit severe performance degradation as video duration increases, while suggesting that context length, image-understanding ability, and LLM backbone choice are key factors for future progress.

Significance. If the observed degradation can be shown to isolate long-range temporal reasoning rather than input-length artifacts, MLVU would supply a useful diagnostic benchmark that highlights concrete limitations in current MLLMs and could steer targeted improvements in context handling and temporal modeling.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (empirical study): the central claim that models exhibit 'severe performance degradation when handling longer videos' is not supported by any description of frame sampling rates, visual-token budgets, or context-window management. Without holding the number of frames per minute and total visual tokens constant while varying only duration, the reported drop cannot be unambiguously attributed to LVU comprehension failures rather than input constraints.
  2. [§3] §3 (benchmark construction): task definitions, metric formulations, statistical significance controls, and video-exclusion criteria are not detailed. These omissions leave the cross-model and cross-duration comparisons only moderately supported and make it difficult to assess whether the chosen tasks genuinely probe long-range reasoning.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'the inappropriateness for evaluating LVU performances' of prior benchmarks is stated without enumerating the specific shortcomings (e.g., length caps, task coverage) that MLVU is designed to remedy.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. The comments highlight important areas where additional methodological transparency is needed to strengthen the claims about performance degradation and the validity of the benchmark tasks. We have revised the manuscript to address both major comments by expanding the relevant sections with the requested details on input processing and task construction. Point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (empirical study): the central claim that models exhibit 'severe performance degradation when handling longer videos' is not supported by any description of frame sampling rates, visual-token budgets, or context-window management. Without holding the number of frames per minute and total visual tokens constant while varying only duration, the reported drop cannot be unambiguously attributed to LVU comprehension failures rather than input constraints.

    Authors: We agree that the original manuscript did not sufficiently document the input preprocessing pipeline, which limits the ability to isolate duration effects. In the revised version we have added a dedicated subsection in §4 describing the uniform frame-sampling strategy (fixed frames per minute across all durations), the per-model visual-token budget constraints, and the context-window truncation policy. With these controls held constant, the degradation trend remains statistically significant; we have included an additional figure and table that replot results under fixed token budgets to make this explicit. These changes directly address the concern and allow readers to attribute the drop more confidently to long-range reasoning limitations. revision: yes

  2. Referee: [§3] §3 (benchmark construction): task definitions, metric formulations, statistical significance controls, and video-exclusion criteria are not detailed. These omissions leave the cross-model and cross-duration comparisons only moderately supported and make it difficult to assess whether the chosen tasks genuinely probe long-range reasoning.

    Authors: We acknowledge that §3 was too concise on these points. The revised manuscript now contains expanded subsections that (1) provide formal definitions and input-output formats for each task, (2) specify the exact metrics (accuracy, mean average precision, or normalized edit distance as appropriate), (3) describe the bootstrapping procedure used for statistical significance, and (4) list the explicit exclusion criteria applied during video curation (e.g., minimum duration, genre balance, and quality filters). These additions make the benchmark construction reproducible and clarify how each task targets long-range temporal dependencies rather than short-term cues. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark proposal with direct external model evaluations

full rationale

The paper introduces the MLVU benchmark and reports direct empirical results from evaluating 23 third-party MLLMs on it. No derivations, equations, fitted parameters, or self-referential claims exist that reduce any reported result to a quantity defined by the authors' own inputs or prior work. The central claims rest on external model performance measurements rather than any internal construction or self-citation chain.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

This is an empirical benchmark proposal paper. It relies on the domain assumption that existing LVU benchmarks are inadequate due to length and diversity limits, with no free parameters, mathematical axioms, or new postulated entities introduced.

assumptions (1)
  • domain assumption Existing video understanding benchmarks are severely constrained by insufficient lengths, lack of diversity in video types and tasks, and inappropriateness for LVU evaluation.
    Directly stated in the abstract as the motivation for creating MLVU.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLVU: Benchmarking Multi-task Long Video Understanding." pith.science (2026). https://pith.science/paper/T7EF4SUO

@misc{pith2026240604264,
  author       = {Pith},
  title        = {Pith review of: MLVU: Benchmarking Multi-task Long Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7EF4SUO}},
  note         = {Machine review of arXiv:2406.04264}
}
read the original abstract

The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark called MLVU (Multi-task Long Video Understanding Benchmark) for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: \textit{1)} The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. \textit{2)} The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. \textit{3)} The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 23 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding ability, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 110 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 110 Pith citations

  1. EgoSound: Benchmarking Sound Understanding in Egocentric Videos

    cs.CV 2026-02 unverdicted novelty 8.0 of 10

    EgoSound is a new benchmark with 7315 QA pairs across seven tasks to evaluate egocentric sound understanding in multimodal large language models.

  2. Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    SFPruner performs redundancy-aware visual token pruning in a single non-iterative pass using semantics-guided ridge leverage scores and directional masking, cutting selection latency from 112.4 ms to 2.5 ms at 512 tok...

  3. VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

    cs.CV 2026-07 conditional novelty 7.0 of 10

    VIABench provides 761 long-form egocentric videos from blind individuals with 14,526 annotations across three assistance tasks, and shows current multimodal LLMs achieve best overall scores below 30.

  4. Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Future-L1 interleaves latent visual spans with text in MLLM decoding, trained on a custom Future-L1-50K dataset via LA-DAPO RL, and reports SOTA gains on FutureBench (61.0 to 85.4) and TwiFF-Bench (2.44 to 3.04).

  5. VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    VideoKR supplies 315K knowledge-intensive video reasoning examples and a dedicated benchmark, with experiments indicating post-training gains on reasoning tasks that require both video content and external knowledge.

  6. Benchmarking Visual State Tracking in Multimodal Video Understanding

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    VSTAT benchmark shows state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines on visual state tracking, failing at visual perception despite correct textual reasoning.

  7. SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SVI-Bench provides 35K hours of sports video with 9 tasks across four cognitive levels, revealing models drop from ~74% on action QA to 5% on agentic evidence integration.

  8. An Efficient Streaming Video Understanding Framework with Agentic Control

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    R3-Streaming uses cascaded control with age-aware memory forgetting and TB-GRPO reinforcement learning to reach SOTA scores of 57.92 on OVO-Bench and 76.36 on StreamingBench with 95-96% fewer visual tokens.

  9. MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MemLens benchmark shows long-context LVLMs lose accuracy with length while memory agents lose visual fidelity, with multi-session reasoning below 30% for most systems and neither approach solving the task alone.

  10. CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    CoRDS selects a compact KV-cache subset via joint-space coreset coverage and log-det diversity to outperform token-wise heuristics on long-video VLM benchmarks.

  11. A Benchmark for Omni-Modal Reasoning in Long Videos

    cs.CV 2025-12 reject novelty 7.0 of 10

    A new 45-minute-scale omni-modal video Q&A benchmark and a training-free retrieval-refine agent, whose reported agent score (66.64% in the abstract) is not supported by the paper's own main results (44.66%).

  12. Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

    cs.CV 2025-06 conditional novelty 7.0 of 10

    MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.

  13. SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning

    cs.CV 2025-06 conditional novelty 7.0 of 10

    SIV-Bench is a new video benchmark with 2,792 clips and 5,455 QA pairs that evaluates MLLMs on social scene understanding, state reasoning, and dynamics prediction using social relation theory.

  14. EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models

    cs.CV 2025-06 conditional novelty 7.0 of 10

    EPFL-Smart-Kitchen-30 is a 29.7-hour multimodal cooking dataset with 60k action segments and four benchmarks, including a kinematic-focused VQA benchmark that shows current video-language models struggle with hand and...

  15. ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding

    cs.CV 2025-05 conditional novelty 7.0 of 10

    ScaleLong embeds four timescale question types into the same long videos, and evaluation of 23 MLLMs reveals a U-shaped accuracy curve across timescales.

  16. Vid-SME: Membership Inference Attacks against Large Video Understanding Models

    cs.CV 2025-05 reject novelty 7.0 of 10

    Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member ...

  17. WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

    cs.CV 2025-02 unverdicted novelty 7.0 of 10

    WorldSense provides the first benchmark requiring synergistic audio-video-text understanding on 1,662 real-world videos and 3,172 QA pairs, where the best current multimodal LLM reaches only 65.1% accuracy.

  18. Online Video Understanding: OVBench and VideoChat-Online

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A new benchmark and a memory-bank architecture let a 4B video AI outperform larger offline and streaming models on online video understanding tasks.

  19. Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

    cs.CV 2026-03 accept novelty 6.5 of 10

    EC-Bench finds best MLLMs score 29.98% enumeration F1 and 23.74% counting accuracy on 152 hour-scale videos, with errors driven by instance identification and temporal grounding rather than arithmetic.

  20. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  21. Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An agent that searches a long video by navigating an adaptive temporal tree with zoom-in/zoom-out/shift actions improves grounded long-video QA, but the headline CG-Bench result is measured on a heavily filtered subset.

  22. FOLIO: Focused Semantic Memory for Streaming Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Entity-centered focus-guided streaming memory lifts Qwen3-VL-8B to 82.0/69.1 Perception/Backward on OVO-Bench and 74.5 on StreamingBench while cutting writer tokens by ~32%.

  23. TimeThink: Reasoning with Time for Video LLMs

    cs.CV 2026-07 accept novelty 6.0 of 10

    TimeThink adds step-wise temporal process rewards (max IoU of referenced intervals) to GRPO for Video-LLMs, improving grounding and reasoning over outcome-only RL baselines.

  24. Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding

    cs.AI 2026-06 conditional novelty 6.0 of 10

    ReMem improves zero-shot long-video QA by combining LLM-based temporal granularity parsing, CLIP-based dual-semantic frame scoring, and structure-aware dynamic frame routing.

  25. From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Audit of four VideoQA benchmarks reveals text-only shortcuts in VLMs; new diagnostics Blind Gap, Visual Gain, and Shortcut Score quantify and filter visual dependence.

  26. DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences

    cs.CV 2026-06 conditional novelty 6.0 of 10

    Rule-reward training on controllable cross-video differences (Grounding + MCQ) improves Video MLLM local spatiotemporal evidence localization and transfers to general video QA benchmarks.

  27. HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.

  28. How Well Can Your Video Model Remember? Measuring Memory-Budget Trade-offs in Long Video Understanding

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Fits a model where logit-accuracy scales linearly in log frame budget B with distance-dependent exponent α(D) that decays log-linearly with temporal distance D, based on 155k binary predictions across ten models.

  29. Harnessing Streaming Video in the Wild

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Presents Streaming-Train-248K dataset, Streaming Harness system, and Streaming-Eval benchmark to enable VLMs for proactive, memory-equipped streaming video understanding.

  30. MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MemoryCard organizes long videos into self-contained topic-aware Memory Cards that improve long-video QA accuracy by up to 21.8% relative under fixed visual-token budgets.

  31. Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    RLVR exhibits correct-set turnover where solved problems regress during training, and a periodic review mechanism exploiting a repair-window principle improves retention and performance over baselines.

  32. MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MOSS-Video-Preview introduces a cross-attention architecture and synthesized real-time QA data to enable continuous perception, answer revision, and faster inference in video-language models compared to decoder-only designs.

  33. TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TeachObs provides human-validated segment- and lesson-level labels on 30 classroom videos and finds frontier VLMs inconsistent, frame-inflated, and over-generous versus expert raters.

  34. DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    DynFrame introduces tokenized learnable span-density retrieval and Segment-Decoupled GRPO in video MLLMs, achieving competitive or SOTA results on six benchmarks with 4B and 8B models.

  35. OProver: A Unified Framework for Agentic Formal Theorem Proving

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    OProver-32B achieves top Pass@32 scores on MiniF2F, ProverBench, and PutnamBench by combining continued pretraining with iterative agentic proving, retrieval, SFT on repairs, and RL on unresolved cases using a 6.86M-p...

  36. Video-Zero: Self-Evolution Video Understanding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Video-Zero is an annotation-free Questioner-Solver co-evolution framework that centers self-evolution on temporally localized evidence to improve video VLMs.

  37. ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    ForestPrune prunes 90% of visual tokens in video MLLMs like LLaVA-OneVision while retaining 95.8% accuracy by modeling tokens as spatial-temporal forests and scoring importance via tree depth and node roles.

  38. Towards Effective Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrieval

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    OneClip-RAG enables MLLMs to handle long videos via one-shot clip retrieval and unified chunking-retrieval, delivering performance gains like matching GPT-5 level on MLVU with high efficiency on standard GPUs.

  39. Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding

    cs.CV 2025-12 conditional novelty 6.0 of 10

    AVP, an iterative plan-observe-reflect agent that samples query-relevant video segments directly, reports state-of-the-art accuracy on five long-video QA benchmarks with roughly 5.4x less inference time than the leadi...

  40. MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.

  41. MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs

    cs.CV 2025-08 reject novelty 6.0 of 10

    MMG-Vid prunes up to 75% of visual tokens in video LLMs with near-lossless accuracy by allocating budgets to segments and selecting temporally novel, spatially diverse tokens.

  42. AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Two drones at different altitudes, one guided by a vision-language model and one by a local navigator, reach targets more often than single-drone baselines on a new UAV navigation benchmark.

  43. HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.

  44. ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

    cs.CV 2025-07 unverdicted novelty 6.0 of 10

    ReGATE introduces a teacher-student adaptive token elision method that reduces training tokens to 38% while matching or exceeding baseline accuracy on multimodal benchmarks.

  45. CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos

    cs.CV 2025-07 conditional novelty 6.0 of 10

    CausalStep introduces a stepwise video QA protocol and reports that top multimodal models (chain success rate 51%) remain far below human performance (79%) on explicit causal chains.

  46. Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new 1,000-video benchmark with adversarial question variants shows video LLMs remain far below human accuracy and robustness on short real-world videos.

  47. DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    DisCo assigns each visual token to a unique concept from the caption and aligns its attention across frames, improving video MLLM accuracy and token efficiency.

  48. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  49. DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DenseWorld-1M provides one million images with detailed object captions, pixel masks, and spatial relations by chaining SAM, APE, RAM++, and VLMs through a three-stage labeling pipeline.

  50. Flash-VStream: Efficient Real-Time Understanding for Long Video Streams

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Flash-VStream uses a two-memory design (clustered low-res summaries plus high-res key-frame details) to answer questions about long videos with state-of-the-art benchmark scores and sub-second latency.

  51. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  52. VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.

  53. MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.

  54. MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    MMR-V is a new video QA benchmark requiring long-range, multi-frame reasoning, on which the best AI model scores 52.5% versus 86% for humans.

  55. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

  56. Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Mixed-R1 uses four reward types under GRPO, including a new bidirectional max-average token similarity (BMAS) reward, and lifts MLLM reasoning benchmarks by 2-5%.

  57. Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Nar-KFC improves long-video question answering by interleaving query-relevant diverse keyframes with captions of non-keyframes, raising accuracy on Video-MME, LongVideoBench, and MLVU across 7B MLLMs.

  58. One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    TrajViT tokenizes videos via panoptic sub-object trajectories, achieving 10x token reduction and outperforming ViT3D by 6% on retrieval and 5.2% on VideoQA tasks with faster training and inference.

  59. VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VCapsBench is a video caption quality benchmark with 109,796 QA pairs across 21 fine-grained dimensions on 5,677 videos, evaluating caption accuracy, inconsistency, and coverage.

  60. TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TUNA introduces a 1,000-video benchmark with dense temporal captions and 1,432 multiple-choice questions, and finds that current video LMMs are weakest at camera motion, action sequences, and multi-subject scenes.

See all 110 Pith citations

Reference graph

Works this paper leans on

63 extracted references · 63 canonical work pages · cited by 110 Pith papers (see all)

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 6, 3

    work page Pith review arXiv 2023

  2. [2]

    Claude 3

    Anthropic. Claude 3. https://www.anthropic.com/ news/claude-3-family, 2024. 7, 2

    work page 2024

  3. [3]

    arXiv preprint arXiv:2404.03413 , year=

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elho- seiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024. 6, 7, 2

    work page arXiv 2024

  4. [4]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    work page Pith review arXiv

  5. [5]

    Frozen in time: A joint video and image encoder for end-to- end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021. 4, 3

    work page 2021

  6. [6]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. 1

    work page 1901

  7. [7]

    arXiv preprint arXiv:2406.04325 , year=

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 7, 2

    work page arXiv 2024

  8. [8]

    How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2, 6, 7

    work page Pith review arXiv 2024

Show all 63 references
  1. [9]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 6, 7, 2

    2024 arXiv

  2. [10]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343–1...

    2022

  3. [11]

    Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023, 2024. 2, 6, 7

    2024

  4. [12]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1, 3

    2023 arXiv

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 1, 2

    2024 arXiv

  6. [14]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171, 2024. 6

    2024

  7. [15]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  8. [16]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726, 2024. 3, 7, 2

    2024

  9. [17]

    Movienet: A holistic dataset for movie understanding

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 709–727. Springer, 2020. 2, 3

    2020

  10. [18]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  11. [19]

    Complex video rea- soning and robustness evaluation suite for video-lmms

    Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Federico Tombari, Fa- had Shahbaz Khan, and Salman Khan. Complex video rea- soning and robustness evaluation suite for video-lmms. arXiv preprint arXiv:2405.03690, 2024. 1

    2024

  12. [20]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706–715, 2017. 6

    2017

  13. [21]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696, 2018. 2, 4

    2018 arXiv

  14. [22]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 1

    2023 arXiv

  15. [23]

    Otter: A multi-modal model 9 with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model 9 with in-context instruction tuning. CoRR, abs/2305.03726,

  16. [24]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6, 7

    2024 arXiv

  17. [25]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 7

    2023 arXiv

  18. [26]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. arXiv preprint arXiv:2311.17005, 2023. 1, 2, 3, 7

    2023

  19. [27]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 1, 2, 3, 7

    2023

  20. [28]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 2, 7

    2023 arXiv

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023. 2, 7

    2023

  22. [30]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024. 3

    2024

  23. [31]

    Lost in the middle: How language models use long con- texts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long con- texts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. 4

    2024

  24. [32]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1

    2023 arXiv

  25. [33]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2

    2023

  26. [34]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36, 2023. 1, 2, 3

    2023

  27. [35]

    Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, page...

    2019

  28. [36]

    Video-bench: A comprehen- sive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehen- sive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023. 1

    2023

  29. [37]

    OpenAI. Gpt-4o. https://openai.com/index/ hello-gpt-4o/, 2024. 6, 7, 2

    2024

  30. [38]

    Retrieving-to-answer: Zero-shot video question answering with frozen large lan- guage models

    Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, and Hongsheng Li. Retrieving-to-answer: Zero-shot video question answering with frozen large lan- guage models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 272–...

    2023

  31. [39]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. arXiv preprint arXiv:2312.02051, 2023. 2

    2023

  32. [40]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485, 2024. 2, 3, 6, 7

    2024

  33. [41]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023. 1, 2, 3, 7, 4

    2023

  34. [42]

    Moviellm: Enhancing long video understanding with ai-generated movies

    Zhende Song, Chenchen Wang, Jiamu Sheng, Chi Zhang, Gang Yu, Jiayuan Fan, and Tao Chen. Moviellm: Enhancing long video understanding with ai-generated movies. arXiv preprint arXiv:2403.01422, 2024. 7, 2

    2024

  35. [43]

    Real-world anomaly detection in surveillance videos

    Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6479–6488, 2018. 4, 1, 2

    2018

  36. [44]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  38. [46]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517,

  39. [47]

    Pax- ion: Patching action knowledge in video-language founda- tion models

    Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Pax- ion: Patching action knowledge in video-language founda- tion models. Advances in Neural Information Processing Systems, 36, 2023. 3

    2023

  40. [48]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2), 2021. 3

    2021

  41. [49]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 2

    2024

  42. [50]

    Next-qa: Next phase of question-answering to explaining tem- poral actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining tem- poral actions. In Proceedings of the IEEE/CVF conference 10 on computer vision and pattern recognition, pages 9777– 9786, 2021. 2, 3

    2021

  43. [51]

    Funqa: Towards surprising video comprehension

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint arXiv:2306.14899, 2023. 3

    2023

  44. [52]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 1, 2, 3

    2016

  45. [53]

    Retrieval-based video language model for efficient long video question answering

    Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-based video language model for efficient long video question answering. arXiv preprint arXiv:2312.04931,

  46. [54]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2, 7

    2023 arXiv

  47. [55]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2019. 4

    2019

  48. [56]

    Activitynet-qa: A dataset for understanding complex web videos via question answer- ing

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answer- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019. 3

    2019

  49. [57]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023. 1, 3, 8

    2023

  50. [58]

    Movie101: A new movie understanding bench- mark

    Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin. Movie101: A new movie understanding bench- mark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), pages 4669–4684, 2023. 2, 6, 1

    2023

  51. [59]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2, 7

    2023 arXiv

  52. [60]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 1, 2, 3, 6, 7

    2024 arXiv

  53. [61]

    ∞ bench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. ∞ bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024. 6

    2024

  54. [62]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. 6

    2023

  55. [63]

    elderlyindividual

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els. In The Twelfth International Conference on Learning Representations, 2023. 2 11 MLVU: Benchmarking Multi-task Long Vi...

    2023

Pith tools

Reviewed May 14, 2026 · model on record in the stance chip above.