REVIEW 22 cited by
VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.
Forward citations
Cited by 22 Pith papers
-
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.
-
EgoExo-Con: Exploring View-Invariant Video Temporal Understanding
Most Video-LLMs answer temporal questions far less consistently when the same event is shown from ego and exo views, and a GRPO variant with a reasoning-similarity reward partially closes the gap.
-
Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
By probing visual, projection, and response representations, the authors find that most VLM visual knowledge loss for recognition and counting occurs in the language decoder, while spatial understanding is lost in the...
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
TCoT uses a single VLM to select question-relevant video frames from segments, then answers from that curated context, improving video QA accuracy across four benchmarks and three VLMs.
-
TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
Weakly supervised vision-language model jointly generating open-ended video QA answers with temporal groundings, reporting SOTA on NExT-GQA, MSVD-QA, and ActivityNet-QA.
-
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.
-
FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understanding
FRAG answers questions about long videos and documents by using an LMM to score each frame or page in isolation, selecting the top-K, and generating the answer from only those frames.
-
Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning
A pipeline that builds entailment trees for answer options and verifies each statement against grounded video fragments consistently improves video and image VLMs on commonsense video QA, and a new LLM-based de-biasin...
-
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.
-
V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.
-
Towards Long Video Understanding via Fine-detailed Video Story Generation
A zero-shot method that turns long videos into hierarchical text descriptions and uses them for retrieval and question answering without any fine-tuning.
-
Progress-Aware Video Frame Captioning
A two-stage model trained on VLM-generated, critic-filtered captions produces frame-level captions that track action progression and outperforms existing VLMs on the new FrameCapEval benchmark.
-
Towards Sparse Video Understanding and Reasoning
A video-QA agent that carries only a structured text summary between rounds beats dense-frame baselines on accuracy while using a handful of frames per video.
-
NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understanding
NoteIt converts instructional videos into interactive notes that preserve chapter and step structure and key visual and verbal information, and users significantly preferred it over a commercial baseline.
-
Moment Sampling in Video LLMs for Long-Form Video QA
Moment sampling uses a text-to-video moment retrieval model to select question-relevant frames, improving long-form VideoQA accuracy by about one to two points over uniform sampling.
-
Frame-Level Captions for Long Video Generation with Complex Multi Scenes
Frame-level captions with per-frame cross-attention and parallel multi-window denoising reduce semantic confusion in long multi-scene video generation in the authors' internal evaluation.
-
VideoMultiAgents: A Multi-Agent Framework for Video Question Answering
A multi-agent video QA framework with independent text, video, and scene-graph agents plus an organizer achieves state-of-the-art zero-shot scores on Intent-QA, the EgoSchema subset, and NExT-QA.
-
ClassComet: Exploring and Designing AI-generated Danmaku in Educational Videos to Enhance Online Learning
AI-written danmaku, combining content and emotion types, can match human comment quality and significantly boost learner engagement and quiz gains in short educational videos.
-
AdaVid: Adaptive Video-Language Pretraining
AdaVid trains video-language encoders whose hidden dimensions can be stripped down at inference time, matching a standard model at half the FLOPs on EgoMCQ.
-
HCQA-1.5 @ Ego4D EgoSchema Challenge 2025
An ensemble of LLMs with confidence filtering and low-confidence re-reasoning reaches 77% accuracy on the EgoSchema benchmark, up from 75% for the prior HCQA system.
-
Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensembles
A training-free ensemble of commercial VLMs with prompt and chain-of-thought engineering reaches 79% on EgoSchema, ranking 2nd in the CVPR 2025 challenge, but the ensemble weights are fit to the test labels.
Discussion (0). Continue with ORCID to comment.