Pith. sign in

REVIEW 8 cited by

TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.19702 v2 pith:LTXSYXSG submitted 2024-10-25 cs.CV cs.AIcs.MM

Xiangyu Zeng , Kunchang Li , Chenting Wang , Xinhao Li , Tianxiang Jiang , Ziang Yan , Songze Li , Yansong Shi
show 5 more authors
Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang
This is my paper · ORCID
classification cs.CVcs.AIcs.MM
keywords videomllmslongtemporaltuningunderstandinggroundedinstruction
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to peform detailed video descriptions with the corresponding time stamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively. In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A video reasoning model learns per question whether to reason aloud or answer directly, improving accuracy by about 3 points over the best adaptive baseline while using about 23% fewer output tokens.

  2. Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    Video-OPD uses on-policy distillation from a frontier teacher to turn sparse episode rewards into dense step-wise signals for more efficient post-training of MLLMs on temporal video grounding.

  3. VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.

  4. EgoExo-Con: Exploring View-Invariant Video Temporal Understanding

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Most Video-LLMs answer temporal questions far less consistently when the same event is shown from ego and exo views, and a GRPO variant with a reasoning-similarity reward partially closes the gap.

  5. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  6. Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A GRPO-based post-training recipe for video LLMs using discrete QA rewards plus continuous temporal IoU rewards with variance-based data selection outperforms SFT and Video-R1.

  7. Continual Video-MLLM Adaptation over Evolving Domains

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A frozen Video-MLLM plus per-domain LoRA expert pools with distribution-aware routing improves domain-incremental video QA from 64.38% to 67.59% average accuracy over the strongest prior method.

  8. Uncertainty-quantified Rollout Policy Adaptation for Unlabelled Cross-domain Temporal Grounding

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Rollout-averaged pseudo labels with variance-based confidence weighting let a GRPO-trained temporal grounding model adapt to an unlabelled target domain from only 100-200 videos.

Pith tools