Pith. sign in

REVIEW 1 major objections 2 minor 55 cited by

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

T0 review · 1 major / 2 minor · reviewed 2026-05-18 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LongVideoBench tests long-context video understanding with referring reasoning on videos up to an hour long.

desk verdict LongVideoBench adds a practical new benchmark for hour-scale video QA with a referring reasoning task, though the claim tying gains strictly to frame capacity rests on cross-model comparisons without tight controls. read the letter →

arxiv 2407.15754 v1 pith:NP77ZS2J submitted 2024-07-22 cs.CV cs.CLcs.LG

Haoning Wu , Dongxu Li , Bei Chen , Junnan Li This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords longvideounderstandingmultimodalbenchmarkreferringreasoninglargemodelsquestionansweringlong-contextevaluationinterleavedlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper creates LongVideoBench to fill the gap in public tests for large multimodal models handling long and rich inputs. It collects thousands of web videos with subtitles and designs questions that require retrieving and reasoning over specific details from referred parts of the video. Human annotators produce over six thousand multiple-choice items across seventeen categories. Tests show that even leading proprietary models like GPT-4o and Gemini-1.5-Pro fall short, open-source models lag further behind, and scores rise only for systems that can take in more frames.

What carries the argument

Referring reasoning, the task in which a question contains a referring query that points to related video contexts called the referred context, forcing the model to locate and reason over the relevant details from that context.

What would settle it

A model that processes only a small number of frames yet matches or exceeds the accuracy of models that ingest many more frames on the full set of 6,678 questions would undermine the reported link between frame capacity and benchmark performance.

Watch

Extended reading notes

Core claim

LongVideoBench supplies 3,763 videos and 6,678 questions that frame the core problem as accurate retrieval and reasoning over detailed multimodal information from long interleaved inputs, using a referring-reasoning task in which each question points to a referred context that the model must then analyze.

Load-bearing premise

The human-annotated questions and video selection process accurately capture long-term multimodal understanding without significant curation biases or gaps in coverage of real-world scenarios.

Editorial extensions

If this is right

  • Proprietary models such as GPT-4o, Gemini-1.5-Pro and GPT-4-Turbo still encounter substantial difficulties on hour-long video inputs.
  • Open-source models display an even wider performance gap than their proprietary counterparts.
  • Benchmark scores rise measurably only when models gain the ability to process additional frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Developers could use the benchmark to measure progress toward systems that retain fine detail across extended video sequences without proportional increases in compute.
  • The design may encourage new architectures that better fuse subtitle text with visual content over long time spans.
  • Similar referring-reasoning formats could be adapted to test long-context understanding in other modalities such as audio or document streams.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The manuscript introduces LongVideoBench, a QA benchmark for long-context interleaved video-language understanding consisting of 3,763 web videos (up to 1 hour) with subtitles across diverse themes and 6,678 human-annotated multiple-choice questions in 17 categories. It defines a 'referring reasoning' task in which each question contains a referring query to a referred video context, requiring models to retrieve and reason over detailed multimodal information from long inputs. Evaluations on proprietary LMMs (GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo) and open-source models show substantial challenges and performance gaps, with results indicating that gains occur only for models able to process more frames.

Significance. If the annotations prove reliable and the task genuinely isolates long-context multimodal reasoning, LongVideoBench would be a valuable addition to the field as one of the largest public benchmarks targeting hour-scale video-language inputs. The human-annotated scale, thematic diversity, and explicit focus on retrieval-plus-reasoning over referred contexts provide a concrete testbed for future long-context LMMs. The reported performance ceilings on current frontier models already supply useful empirical signals.

major comments (1)
  1. [Abstract and Results] Abstract and Results section: the statement that 'model performance on the benchmark improves only when they are capable of processing more frames' rests on cross-model comparisons. These models differ simultaneously in scale, pre-training corpus, instruction tuning, and long-context adaptation; no within-model ablation that holds architecture and training fixed while varying only frame count or context length is described. The causal 'only when' phrasing therefore lacks direct support and risks confounding.
minor comments (2)
  1. [Benchmark Construction] Benchmark construction section: inter-annotator agreement statistics, question validation procedures, and explicit exclusion criteria for the 6,678 questions are not reported in detail. Adding these would strengthen the claim that the questions comprehensively require long-term multimodal understanding.
  2. [Task Definition] The paper positions 'referring reasoning' as a novel formulation, yet the distinction from prior referring-expression or long-video QA tasks could be made more explicit to clarify its incremental contribution.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and constructive comment on the interpretation of our results. We address the major comment below and will make the corresponding revisions to the manuscript.

read point-by-point responses

  1. Referee: [Abstract and Results] Abstract and Results section: the statement that 'model performance on the benchmark improves only when they are capable of processing more frames' rests on cross-model comparisons. These models differ simultaneously in scale, pre-training corpus, instruction tuning, and long-context adaptation; no within-model ablation that holds architecture and training fixed while varying only frame count or context length is described. The causal 'only when' phrasing therefore lacks direct support and risks confounding.

    Authors: We agree with the referee that the original phrasing in the abstract and results section implies a stronger causal relationship than is warranted by the cross-model comparisons presented. Our evaluations show that models with longer effective context windows (such as Gemini-1.5-Pro) achieve higher accuracy, while others plateau, but we acknowledge that these models also differ in scale, training data, and other factors. We will revise the abstract to replace the causal 'improves only when' with a more precise observational statement, e.g., 'we observe that performance on LongVideoBench is higher for models capable of processing more frames.' In the results section we will add explicit discussion of the limitations of cross-model analysis and note that within-model ablations varying only frame count or context length are left for future work. These changes will be incorporated in the revised manuscript. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark is externally constructed and evaluated

full rationale

The paper constructs LongVideoBench from web-collected videos and human-annotated questions in a referring-reasoning task. All reported results are empirical evaluations of independent external models (GPT-4o, Gemini-1.5-Pro, open-source LMMs) on this fixed benchmark. No equations, fitted parameters, self-citations, or derivations are present that reduce any claim to the paper's own inputs by construction. The statement that performance improves only with greater frame capacity is an observational finding from cross-model comparisons, not a self-referential prediction or renamed input.

Assumptions & free parameters 0 free parameters · 0 assumptions · 1 invented entities

This is an empirical benchmark paper with no mathematical derivations, so it introduces no free parameters, standard axioms, or invented physical entities. The primary addition is the curated dataset and task definition.

invented entities (1)
  • referring reasoning task
    purpose: To require models to retrieve and reason over specific multimodal details referenced in the question from long video inputs
    This is a formulated evaluation task introduced for the benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding." pith.science (2026). https://pith.science/paper/NP77ZS2J

@misc{pith2026240715754,
  author       = {Pith},
  title        = {Pith review of: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NP77ZS2J}},
  note         = {Machine review of arXiv:2407.15754}
}
read the original abstract

Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 55 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A video reasoning model learns per question whether to reason aloud or answer directly, improving accuracy by about 3 points over the best adaptive baseline while using about 23% fewer output tokens.

  2. HumanMoveVQA: Can Video MLLMs reason about human movement in videos?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    HumanMoveVQA is a new benchmark that generates 10K+ QA pairs from 3D-lifted video tracks to evaluate video MLLMs on global human trajectory and orientation reasoning.

  3. An Efficient Streaming Video Understanding Framework with Agentic Control

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    R3-Streaming uses cascaded control with age-aware memory forgetting and TB-GRPO reinforcement learning to reach SOTA scores of 57.92 on OVO-Bench and 76.36 on StreamingBench with 95-96% fewer visual tokens.

  4. MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MemLens benchmark shows long-context LVLMs lose accuracy with length while memory agents lose visual fidelity, with multi-session reasoning below 30% for most systems and neither approach solving the task alone.

  5. A Benchmark for Omni-Modal Reasoning in Long Videos

    cs.CV 2025-12 reject novelty 7.0 of 10

    A new 45-minute-scale omni-modal video Q&A benchmark and a training-free retrieval-refine agent, whose reported agent score (66.64% in the abstract) is not supported by the paper's own main results (44.66%).

  6. MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A 500-video benchmark with six reasoning categories shows state-of-the-art vision-language models scoring below 25%, with particularly low performance on planning and abstract reasoning.

  7. ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding

    cs.CV 2025-05 conditional novelty 7.0 of 10

    ScaleLong embeds four timescale question types into the same long videos, and evaluation of 23 MLLMs reveals a U-shaped accuracy curve across timescales.

  8. LVBench: An Extreme Long Video Understanding Benchmark

    cs.CV 2024-06 accept novelty 7.0 of 10

    LVBench is a new benchmark for extreme long video understanding that evaluates multimodal large language models on hour-scale videos using tasks designed to probe extended memory and comprehension.

  9. Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An execution-centric training recipe that specializes a vision-language model into four capability experts and consolidates them via weight merging and distillation yields a single embodied model with broad benchmark gains.

  10. The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...

  11. CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CLIP-CC-Bench is a 200-clip benchmark with expert paragraph references that ranks 17 video-language models via an ensemble of five embedding-based semantic judges.

  12. CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.

  13. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  14. MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    MedStreamBench integrates 22 medical datasets into 5,419 QA instances across retrospective, present, future, and proactive temporal settings to evaluate streaming and proactive medical video understanding.

  15. EFlow: Learning Evidence Flow for Long-Video Reasoning with Adaptive Reflection

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    Separating temporal grounding from answer reasoning, plus low-confidence full-video re-reading, modestly improves long-video QA on Qwen3-VL across five benchmarks.

  16. From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Audit of four VideoQA benchmarks reveals text-only shortcuts in VLMs; new diagnostics Blind Gap, Visual Gain, and Shortcut Score quantify and filter visual dependence.

  17. DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences

    cs.CV 2026-06 conditional novelty 6.0 of 10

    Rule-reward training on controllable cross-video differences (Grounding + MCQ) improves Video MLLM local spatiotemporal evidence localization and transfers to general video QA benchmarks.

  18. HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.

  19. Harnessing Streaming Video in the Wild

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Presents Streaming-Train-248K dataset, Streaming Harness system, and Streaming-Eval benchmark to enable VLMs for proactive, memory-equipped streaming video understanding.

  20. MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MOSS-Video-Preview introduces a cross-attention architecture and synthesized real-time QA data to enable continuous perception, answer revision, and faster inference in video-language models compared to decoder-only designs.

  21. PEEK: Picking Essential frames via Efficient Knowledge distillation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    PEEK distills caption-conditioned frame relevance into a lightweight visual model, outperforming adaptive baselines on ActivityNet Captions and MSR-VTT especially at 1-2 frame budgets while adding only 5.2% overhead.

  22. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.

  23. TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TeachObs provides human-validated segment- and lesson-level labels on 30 classroom videos and finds frontier VLMs inconsistent, frame-inflated, and over-generous versus expert raters.

  24. Prism: Spectral-Aware Block-Sparse Attention

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Mean-pooling under RoPE attenuates high-frequency position signals (a spectral blind spot), and Prism's dual-band RMS-calibrated block selection recovers enough signal to match full attention at up to 5.1× prefill speedup.

  25. VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.

  26. Position: Reasoning After Perception Means Reasoning Without Vision

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.

  27. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  28. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

  29. VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VCapsBench is a video caption quality benchmark with 109,796 QA pairs across 21 fine-grained dimensions on 5,677 videos, evaluating caption accuracy, inconsistency, and coverage.

  30. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  31. QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A co-designed parallel decoder and KV-pruned group prefill pipeline speeds up long video LLM inference by over 3x, from 69.7 to 20.0 seconds on a 30-minute input.

  32. LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    LiveVLM introduces VSB and PaR to compress and retrieve KV cache in streaming video LLMs, enabling LLaVA-OneVision to reach SOTA accuracy among training-free query-agnostic and training-based online models.

  33. Temporal Preference Optimization for Long-Form Video Understanding

    cs.CV 2025-01 conditional novelty 6.0 of 10

    TPO trains video-LMMs to prefer answers generated from complete, relevant frames over answers from incomplete or irrelevant frames, improving temporal grounding on LongVideoBench, MLVU, and Video-MME.

  34. MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

    cs.CV 2025-01 unverdicted novelty 6.0 of 10

    MotionBench is a new benchmark showing poor fine-grained motion understanding in VLMs and proposes TE Fusion to improve performance with higher frame rates.

  35. VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling

    cs.CV 2024-12 unverdicted novelty 6.0 of 10

    VideoChat-Flash applies hierarchical video token compression to achieve ~50x reduction in context length for long videos while maintaining near-original performance on long-context benchmarks.

  36. GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Giraffe extends Qwen-VL and Qwen2-VL to 128K context using a new multimodal RoPE extension, a curated data recipe, and hybrid-resolution inference, reaching state-of-the-art results among open-source long-context VLMs.

  37. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  38. TuringViT: Making SOTA Vision Transformers Accessible to All

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    TuringViT uses Turing Linear Attention, VISTA-Curation, and dynamic-resolution pretraining to outperform open ViT baselines with 10% data while improving VLM performance and high-resolution latency.

  39. VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    VisionPulse is a step-wise visual token pruning method for LMMs that retains 5% of tokens per step, shortens reasoning traces by 11.2%, and maintains accuracy.

  40. EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    EgoCoT-Bench provides 3,172 verifiable QA pairs across perception, anticipation, and reasoning tasks on egocentric videos, revealing that many MLLMs give answer-correct but evidence-inconsistent explanations.

  41. Omnidirectional Spatial Modeling from Correlated Panoramas

    cs.CV 2025-09 conditional novelty 5.0 of 10

    The authors create a cross-frame panoramic VQA benchmark from 3D scene data and show that GRPO fine-tuning of Qwen2.5-VL raises its score on that benchmark.

  42. Task-Aware KV Compression For Cost-Effective Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.

  43. Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysis

    cs.CV 2025-06 conditional novelty 5.0 of 10

    VideoLLaMA2's tennis sequence edit score jumps from 39.7 to 76.0 when text coordinates from detection models are included in the prompt, and a separately fine-tuned CLIP encoder raises single-event accuracy from 0.41 to 0.56.

  44. Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.

  45. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  46. Qwen2.5-VL Technical Report

    cs.CV 2025-02 unverdicted novelty 5.0 of 10

    Qwen2.5-VL reports a vision-language model family using native dynamic-resolution ViT and absolute time encoding that matches GPT-4o on document and diagram tasks while supporting hour-long videos with second-level lo...

  47. Ola: Pushing the Frontiers of Omni-Modal Language Model

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Ola, a 7B omni-modal language model, achieves competitive image, video, and audio understanding with progressive modality alignment, though it does not beat all specialized models on every benchmark.

  48. VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos

    cs.IR 2025-02 conditional novelty 5.0 of 10

    VideoRAG combines graph-based text indexing with multimodal visual embeddings to answer questions across multi-hour video collections, supported by a new 134-hour benchmark.

  49. TinyLLaVA-Video: Towards Smaller LMMs for Video Understanding with Group Resampler

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A small video model with a group resampler reduces video input to a few hundred tokens and beats several 7B models on common video benchmarks.

  50. FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filtering

    cs.CV 2024-12 conditional novelty 5.0 of 10

    FocusChat uses prompt-guided spatial and temporal filtering to cut visual tokens to as few as 16 while matching or beating a larger Video-LLaMA baseline.

  51. mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

    cs.CV 2024-08 unverdicted novelty 5.0 of 10

    mPLUG-Owl3 introduces hyper attention blocks to integrate vision and language for long image-sequence understanding and reports SOTA results on single-image, multi-image, and video benchmarks.

  52. InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning

    cs.CV 2026-06 unverdicted novelty 4.0 of 10

    InternVideo3 introduces Multimodal Contextual Reasoning and M^2LA attention to enable closed-loop evidence accumulation in long-video understanding and agentic tool use, reporting strong benchmark results.

  53. DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DynTok dynamically merges similar adjacent visual tokens into groups, reducing video token counts to 44.4% with comparable or better video understanding accuracy.

  54. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  55. Infinite Video Understanding

    cs.CV 2025-07 conditional novelty 3.0 of 10

    The paper argues that video understanding research should aim at processing streams of arbitrary, unbounded duration and outlines the challenges, directions, and metrics needed.

Reference graph

Works this paper leans on

76 extracted references · 76 canonical work pages · cited by 55 Pith papers

  1. [1]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    URL https://huggingface.co/blog/idefics. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El ...

    work page Pith review arXiv 2023

  2. [2]

    For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Sec. C (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. D (d) Have you read the ethics review guidelines and ensur...

    work page

  3. [3]

    (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]

    If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]

    work page

  4. [4]

    for benchmarks)

    If you ran experiments (e.g. for benchmarks)... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] All code, data and instructions can be assessed at https://longvideobench.github.io. (b) Did you specify all the training details (e.g., data split...

    work page

  5. [5]

    If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [Yes] (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] All assets can be assessed at https://longvide...

    work page

  6. [6]

    type": "image_url

    If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] The instructions are included separately in Sec. E.1. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applica...

    work page 2024

  7. [7]

    Find an action or event

    work page

  8. [8]

    Pause, describe/outline the scene information as the question stem

    work page

Show all 76 references
  1. [9]

    Use this action or event as the answer

  2. [10]

    S CENE -REFERRED OBJECT (S2O)

    You may refer to these examples: • What is the boy in the video doing at Danube Square? • What happens after all the ingredients are placed in the pot? • When the video transitions to the office, what are the employees doing? • What are the characters in the video doing in the...

  3. [12]

    Describe/outline the scene information as the question stem

  4. [13]

    Use the appearing people/objects and the absent ones as correct and incorrect answers respectively

  5. [14]

    S CENE -REFERRED OBJECT ATTRIBUTE (S2A)

    You may refer to these examples: • What objects appeared in Laura’s bedroom in the video? (Lifestyle) • When all the ingredients are chopped and placed together, which ingredient did not appear? (Cooking) • Which communication method was not mentioned in the fourth section? (P...

  6. [15]

    Find a scene, observe the people or objects in this scene

  7. [16]

    Describe/outline the scene information and determine an object as the question stem

  8. [17]

    Use existing and non-existing attributes of the object as correct and incorrect answers respectively, such as material, color, shape, transparency, surface characteristics, structural features

  9. [18]

    EVENT -REFERRED OBJECT (E2O)

    You may refer to these examples: • What clothes is Laura wearing in the bedroom with an air conditioner, a bed, and a clothes rack? • What color is used to represent the feed forward layer in the Transformer network in Figure 4? • Is the person in red clothing wearing glasses ...

  10. [19]

    Find an action or event. 18

  11. [20]

    Identify the participating people or objects

  12. [21]

    Describe this action/event as the question stem

  13. [22]

    The options should also be as detailed as possible

    Based on the subtitles at the time of the action/event or other background information, detail the participating people/objects as the answer. The options should also be as detailed as possible

  14. [23]

    OBJECT -REFERRED EVENT (O2E)

    You may refer to these examples: • Who participated in and won the duel in the movie? • Which character finished knitting the sweater? • What object exploded in the chemistry experiment in the video? • What is the expression of the input variable passed into the Transformer in...

  15. [24]

    Find a person or object

  16. [25]

    Identify the actions/events that happens at their appearance

  17. [26]

    Describe the person/object as the question stem

  18. [27]

    Based on a scene where this person/object appears (e.g., first appearance), ask what event happened or what action they took at that time

  19. [28]

    T EXT-REFERRED EVENT (T2E)

    You may refer to these examples: • What did the girl in red do the first time she appeared? • What happened the first time a volcano appeared in the video? VI. T EXT-REFERRED EVENT (T2E)

  20. [30]

    Identify the action in the current frame of the video

  21. [31]

    Think of a few actions that did not appear in the video but are easily confused

  22. [32]

    Use the action from step 2 as the correct answer, and the actions from step 3 as other options

  23. [33]

    bidirectional encoder

    You may refer to these examples: • What was the protagonist doing when mentioning the Renaissance? • What event happened when “bidirectional encoder” first appeared in the subtitles? VII. T EXT-REFERRED OBJECT (T2O)

  24. [35]

    Identify a certain object in the frame; for example, a black water bottle

  25. [36]

    Think of a few objects that did not appear in the video but are easily confused, such as a red water bottle, a black hat, a water dispenser, a transparent water cup

  26. [37]

    Use the object from step 2 as the correct answer, and the objects from step 3 as other options

  27. [38]

    revolutionary changes

    You may refer to these examples: • What object was present when the lecturer mentioned “revolutionary changes”? • Which object did not appear when talking about Jack and Rose having a heart-to-heart conversation? VIII. T EXT-REFERRED OBJECT ATTRIBUTE (S2A)

  28. [39]

    Find a segment of subtitles, pause the video

  29. [40]

    Identify a certain object in the frame

  30. [41]

    Identify an attribute of the object, such as material, color, shape, transparency, surface characteristics, structural features

  31. [42]

    Use the object from step 2 as the correct answer, and the attributes from step 3 as other options. 19

  32. [43]

    The specific instructions for each category of (L2) questions are as follows

    You may refer to these examples: • What was Tesla’s hairstyle like when he was mentioned to have invented alternating current? • What color hat was the female protagonist wearing when talking about taking a break? Instructions for (L2) Relation questions. The specific instruct...

  33. [44]

    Find two or more adjacent actions or events

  34. [45]

    Describe one of the actions/events as the question stem, and the other as the correct answer

  35. [46]

    O BJECT BEFORE /AFTER OBJECT (O3O)

    You may refer to these examples: • What did Clara do before taking a photo? (applicable to movie or lifestyle videos) • What needs to be done after installing the screws? (applicable to guide videos) • Which of the following historical/geographical events was mentioned first? ...

  36. [47]

    Find two or more people/objects/concepts that appear in the video

  37. [48]

    Describe one of the objects as the question stem, and the other as the correct answer

  38. [49]

    S EQUENCE OF SCENES (SSS)

    You may refer to these examples: • After Jack appears, which character appears first in this movie? • Which concept is introduced first in the video after entropy is introduced? XI. S EQUENCE OF SCENES (SSS)

  39. [50]

    Find multiple scenes (at least three) in the video

  40. [51]

    Ask questions about the order of these scenes

  41. [52]

    Answer with the correct sequence and use a few scrambled sequences as distractors

  42. [53]

    First, a segment of the experiment video is played, then slides with text are shown, and finally XXXX

    You may refer to this example: • Which of the following scene sequences is correct? • A. First, a segment of the experiment video is played, then slides with text are shown, and finally XXXX. • B. First, slides with text are shown, ... XII. S CENE -REFERRED OBJECT TRACKING (SOS)

  43. [56]

    Then ask in which other scenes did they appear

  44. [57]

    Distractors are scenes where this object did not appear

  45. [58]

    Square on a sunny day, – B

    You may refer to these examples: • In which of the following places did the boy who was running at the beginning of the video appear? – A. Square on a sunny day, – B. On a boat at sea, – C. In a bar on a rainy day, ... • In which other scenes did the protagonist’s lightsaber, ...

  46. [59]

    Find a specific person/object/concept that appears in multiple scenes

  47. [61]

    Then describe another scene and ask what attribute of this person/object/concept has changed at that time

  48. [62]

    Changed from a white T-shirt to a black vest – B

    You may refer to these examples: • What did the boy running at the beginning of the video change into when climbing the mountain at the end? – A. Changed from a white T-shirt to a black vest – B. Changed from red shoes to white shoes – C. ... • What changed in the color of the...

  49. [63]

    Find a segment of subtitles, and an action/event in the video that happens before/after it

  50. [64]

    Rephrase/outline the subtitle as the given information and design the question stem, with the action/event as the correct answer

  51. [65]

    Distractors are other actions/events in the video that do not meet the sequence relationship in the question stem

  52. [66]

    I eat an apple every day

    You may refer to these examples: • What did Clara do after she said, “I eat an apple every day”? • What happened before the narrator mentioned the experiment starting? • What action was performed after the chef said, “Now wait until the steak surface turns golden”? XV . OBJECT...

  53. [67]

    Find the scene where a specific person/object first appears

  54. [68]

    Then find subtitles before or after this timeframe, rephrase/outline the subtitle as the given information and design the question stem, with the object/person as the correct answer

  55. [69]

    Distractors are other people/objects in the video that do not meet the sequence relationship in the question stem

  56. [70]

    100 years later

    You may refer to these examples: • Which characters appeared after the commentary mentioned “100 years later”? • Which animal appeared on screen before mentioning “dietary habits of North American squirrels”? XVI. T EXT-REFERRED OBJECT TRACKING (TOS)

  57. [71]

    Find a specific person/object/concept that appeared at least once along with subtitles

  58. [73]

    Ask on a subtitle at the object’s appearance

  59. [74]

    Distractors are subtitles where this object did not appear at the corresponding moment

  60. [75]

    T EXT-REFERRED OBJECT ATTRIBUTE CHANGE (TAA)

    You may refer to these examples: • With which subtitles did the boy running at the beginning of the video appear? • During which of the following dialogues did the protagonist’s lightsaber, used in the opening fight, appear on screen? XVII. T EXT-REFERRED OBJECT ATTRIBUTE CHANGE (TAA)

  61. [76]

    21 Figure 6: The annotation interface for L ONG VIDEO BENCH

    Find a specific person/object/concept that appeared at least once along with subtitles. 21 Figure 6: The annotation interface for L ONG VIDEO BENCH

  62. [77]

    Define this person/object/concept by their action/attribute in one of the scenes

  63. [78]

    Ask what attribute has changed when XX text is mentioned

  64. [79]

    I am going to sleep

    You may refer to these examples: • What change occurred to the girl in the blue jacket and black hood in the middle of the video when mentioning “I am going to sleep”? – A. She changed the color of her hood – B. She changed into a black jacket – C. She took off her hood – D. S...

    2024

  65. [80]

    Participate in our mandatory training to understand the guidelines of annotation

  66. [81]

    Each annotation includes the following terms: (a) A question; (b) One or more timestamp(s) on the question; (c) Four to five options; (d) A checkbox to pick the correct option

    Watch videos, and provide annotations on these videos. Each annotation includes the following terms: (a) A question; (b) One or more timestamp(s) on the question; (c) Four to five options; (d) A checkbox to pick the correct option

  67. [82]

    Check the correctness of annotations from other annotators

  68. [83]

    raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw

    Report videos that are not appropriate during the process. Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or othe...

Pith tools

Reviewed May 18, 2026 · model on record in the stance chip above.