Pith. sign in

REVIEW 2 major objections 4 minor 76 references

Video LLMs reason better when each reasoning step is scored by how well its time interval matches the evidence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 08:58 UTC pith:UBHRR42J

load-bearing objection Clean process-reward idea for video CoT that actually moves both grounding and open-ended numbers; the auto-annotation caveat is real but does not sink the paper. the 2 major comments →

arxiv 2607.05089 v1 pith:UBHRR42J submitted 2026-07-06 cs.CV

TimeThink: Reasoning with Time for Video LLMs

Handong Li , Longteng Guo , Zikang Liu , Dongze Hao , Yepeng Tang , Zijia Zhao , Jie Jiang , Zhiwei Jin
show 3 more authors
Chen Chen Haonan Lu Jing Liu
This is my paper
classification cs.CV
keywords Video Large Language Modelsreinforcement learningprocess rewardtemporal groundingvideo reasoningtemporal clue stepsGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video models that reason over long clips usually get a reward only for the final answer. That leaves the intermediate search for the right moments unguided, so the model can invent vague timelines or latch onto global cues. TimeThink instead forces every intermediate step to name a candidate time span and then scores that span with a simple temporal overlap (IoU) against automatically recovered evidence segments. The resulting process reward is combined with ordinary answer correctness inside a joint reinforcement-learning objective. With a 20 K automatically annotated training set, the method lifts both localization accuracy and multi-step reasoning on seven standard benchmarks, outperforming other open-source video RL models trained on the same backbone. The practical claim is that teaching a model where to look while it thinks is more effective than teaching it only what the right answer is.

Core claim

Treating each intermediate reasoning step as a temporal clue that must reference a concrete video interval, and rewarding that interval by its maximum IoU with ground-truth evidence segments, produces reasoning trajectories that are both more temporally faithful and more accurate on final-answer metrics than outcome-only reinforcement learning.

What carries the argument

The step-wise temporal process reward: for every clue interval p_k the model emits, R_clue^(k) = max_g IoU(p_k, g) over the set of automatically derived evidence segments; this scalar is normalized into a token-level advantage and added (with weight λ) to the global outcome advantage inside the GRPO update.

Load-bearing premise

The automatically recovered time intervals (scene cuts scored by a large teacher model) are accurate enough to serve as reliable ground truth for the IoU process reward.

What would settle it

Train an otherwise identical model on the same 20 K set but with deliberately scrambled or randomly shifted evidence intervals; if the reported gains on Charades-STA, NExT-GQA and VideoMMMU disappear, the process-reward claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes TimeThink, a GRPO-based RL framework for Video-LLMs that treats intermediate reasoning as a sequence of temporal clue steps, each referencing a candidate video interval p_k. It defines a step-wise process reward R_clue^(k) = max_g IoU(p_k, g) against automatically derived evidence segments E_gt (Eq. 4), combines it with outcome (answer + format) rewards via token-level advantages (Eq. 7, λ=0.5), and trains in two stages on the new TimeThink-RFT-20K dataset (PySceneDetect + Qwen3-VL-235B relevance scoring) followed by broader LLaVA-Video + timestamp data. Experiments on grounding (Charades-STA, CGBench, NExT-GQA), reasoning (VideoMMMU, VSIBench, MMVU), and general understanding benchmarks report consistent gains over same-backbone SFT/GRPO baselines and SOTA among open-source video RL models (Tables 1–4), supported by ablations on reward type, stages, data scale, and λ.

Significance. If the automatic E_gt segments are sufficiently reliable, TimeThink supplies a scalable, process-level temporal credit-assignment signal that prior outcome-only video RL methods lack, yielding measurable improvements in both localization precision and open-ended reasoning without dense human process labels. The work is strengthened by same-backbone controls, multi-benchmark evaluation under lmms-eval, reward-variant and data-scaling ablations (Table 5, Fig. 3), qualitative trajectory comparisons, planned code/model/dataset release, and a lightweight two-stage recipe that first installs grounded behavior then generalizes. These elements make the contribution concrete and reusable for the growing line of video RL research.

major comments (2)
  1. [Sec. 5 / Eq. (4)] Sec. 5 and Eq. (4): The entire process-reward signal is defined by max-IoU against E_gt segments obtained automatically (PySceneDetect shot cuts scored by Qwen3-VL-235B relevance ≥ threshold). No human agreement, precision/recall, or error analysis of these segments is reported (only a small annotator-size ablation in Supp. Table 6). Because the largest claimed lifts (e.g., Charades-STA mIoU +5.0 from (a) to (b) in Table 5, Stage-1-only gains) rest on this proxy, systematic teacher bias (over-segmentation, preference for long clips, multi-event hallucination) would mis-specify R_clue and weaken the causal attribution of gains to “temporal evidence discovery.” A modest human validation study or quantitative characterization of E_gt noise is needed to underwrite the central claim.
  2. [Sec. 4.2 / Table 5] Sec. 4.2–4.3 and Table 5: The process reward evaluates only the referenced interval, never the textual content of the clue step. While this design choice is intentional for flexibility, it leaves open the possibility that the policy learns to emit correctly timed but factually incorrect observations (reward hacking of the temporal channel alone). The binary and mIoU reward variants already exhibit hacking; an analogous content-consistency check or qualitative audit of step correctness (beyond the single NExT-GQA example in Fig. 4) would confirm that the Max-IoU formulation truly improves reasoning fidelity rather than merely temporal formatting.
minor comments (4)
  1. [Sec. 4.3] Eq. (7): The mapping k(t) that assigns tokens inside the <think> block to individual clue steps is not specified algorithmically (regex on time stamps? sequential order?). A short clarifying sentence or pseudocode would aid reproducibility.
  2. [Fig. 1] Fig. 1 and Fig. 2 captions are dense; the right-hand panels of Fig. 1 mix three claims (faithful reasoning, faster convergence, stronger understanding) without corresponding quantitative panels in the main text. Moving the convergence curve into the main body would better support the “faster outcome convergence” claim.
  3. Throughout: minor typesetting artifacts remain (e.g., “TimeThink-RFT-20K” sometimes split, missing spaces around math, “R(k)clue” rendering). A final pass would improve readability.
  4. [Sec. 2.2] Related Work (Sec. 2.2) correctly positions against Video-R1, TimeZero, TempSamp-R1; a one-sentence contrast on why process (vs. outcome-only IoU) transfers better to open-ended QA would sharpen the novelty claim.

Circularity Check

0 steps flagged

No circularity: process reward is an external IoU signal against auto-derived segments; outcome rewards and benchmark metrics are independent of that construction.

full rationale

The paper's central claim is an empirical RL result: adding a step-wise temporal process reward R_clue^(k) = max_g IoU(p_k, g) (Eq. 4) to GRPO, where E_gt comes from an automatically constructed dataset (PySceneDetect + Qwen3-VL-235B relevance scoring, Sec. 5), improves localization and reasoning metrics on held-out external benchmarks (Tables 1-4). The reward definition does not embed the final answer correctness or the benchmark scores; those remain separate outcome terms (R_ans + R_fmt) and independent evaluation protocols (lmms-eval). There is no parameter fitted to a subset of the reported test metrics and then re-presented as a prediction, no uniqueness theorem imported from overlapping authors that forces the method, and no renaming of a known empirical pattern as a first-principles derivation. The two-stage schedule (Stage 1 with process reward, Stage 2 with outcome only) and the ablations (Table 5) are ordinary experimental controls, not tautologies. The automatic nature of E_gt is a validity/assumption risk, not a circular reduction of the claimed gains to their own inputs. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on standard RL and video-LLM machinery plus a small set of design choices (max-IoU process reward, λ = 0.5, automatic teacher annotations). No new physical entities; the invented constructs are methodological.

free parameters (3)
  • process-reward weight λ = 0.5
    Controls strength of step-wise temporal advantage relative to global outcome advantage; default 0.5, ablated in [0.1,1.0] (Table 7).
  • teacher relevance threshold
    Clips scored by Qwen3-VL-235B above an (unspecified numeric) threshold become ground-truth evidence segments for R_clue (Sec. 5).
  • GRPO group size / learning rate / batch size = 8 / 1e-6 / 64
    Standard RL hyper-parameters (group 8, lr 1e-6, batch 64) that affect optimization trajectory but are not claimed as theoretical constants.
axioms (3)
  • ad hoc to paper Temporal IoU between a referenced interval and teacher-derived evidence segments is a valid scalar proxy for the quality of an intermediate reasoning step.
    Core design choice of R_clue (Eq. 4); justified by ablation against binary and mean-IoU variants but not derived from first principles.
  • domain assumption Group Relative Policy Optimization with token-level advantages is a stable and effective RL algorithm for 7 B Video-LLMs.
    Inherited from DeepSeekMath / recent video-RL literature; used without re-derivation.
  • domain assumption Scene-boundary clips produced by PySceneDetect plus a large VLM teacher yield sufficiently accurate evidence segments for process supervision.
    Sec. 5 construction; sensitivity to teacher size is ablated (Table 6) but absolute annotation quality is not human-validated.
invented entities (2)
  • temporal clue step no independent evidence
    purpose: Atomic optimization unit: a free-form reasoning sentence that must reference a concrete time interval p_k = (t_start, t_end).
    Defined in Sec. 3–4; the entire process-reward machinery is built around this construct.
  • TimeThink-RFT-20K no independent evidence
    purpose: Training set of ~20 K QA pairs with automatically attached temporal evidence segments used to compute R_clue.
    Constructed in Sec. 5; enables scalable process rewards without dense human annotation.

pith-pipeline@v1.1.0-grok45 · 24775 in / 2711 out tokens · 32531 ms · 2026-07-11T08:58:40.754231+00:00 · methodology

0 comments
read the original abstract

Video reasoning requires models to identify and verify temporally localized evidence within long video sequences. Recent Video Large Language Models (Video-LLMs) have shown promising reasoning abilities when aligned with reinforcement learning, yet existing approaches typically rely on outcome-based rewards that supervise only the final prediction. Such supervision provides limited guidance on how models should discover the relevant temporal evidence during intermediate reasoning. In this work, we propose TimeThink, a reinforcement learning framework that explicitly guides temporal evidence discovery in Video-LLMs. Our key idea is to treat temporal clue steps as the fundamental optimization primitive of video reasoning, where each reasoning step references a candidate time interval in the video. We introduce a step-wise temporal process reward that provides localized credit assignment for these clues and a joint process--outcome optimization objective that balances reasoning fidelity with task correctness. To enable scalable training, we construct TimeThink-RFT-20K, a dataset with automatically derived temporal evidence segments. Extensive experiments across video reasoning, temporal grounding, and general video understanding benchmarks show that TimeThink consistently improves both temporal localization and reasoning performance, achieving state-of-the-art results among open-source video RL models.

Figures

Figures reproduced from arXiv: 2607.05089 by Chen Chen, Dongze Hao, Handong Li, Haonan Lu, Jie Jiang, Jing Liu, Longteng Guo, Yepeng Tang, Zhiwei Jin, Zijia Zhao, Zikang Liu.

Figure 1
Figure 1. Figure 1: TimeThink vs. Traditional Video-CoT. (Left) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of TimeThink. The framework processes input video and textual queries through a Video-LLM. During the reinforcement learning phase, the model generates an intermediate reasoning trajectory, which is supervised by a step￾wise process reward alongside outcome-based rewards to optimize the policy πθ. where y think denotes the reasoning process enclosed in a <think> block and y ans represents the … view at source ↗
Figure 3
Figure 3. Figure 3: Data Scaling Gains. Rela￾tive performance improvements achieved across multiple benchmarks when scaling the Stage 1 data from 1K to 20K, evalu￾ated after identical Stage 2 training. Performance on Temporal Grounding. As shown in Tab. 3, incorpo￾rating IoU-based rewards for video clues during the intermediate reasoning pro￾cess significantly enhances zero-shot grounding capabilities. On Charades-STA, our me… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparisons on grounded QA. The GRPO baseline yields an incorrect answer with an unfaithful reasoning trajectory. In contrast, TimeThink provides the correct answer while achieving a significantly higher mIoU for query￾grounded segments, supported by a detailed and precise reasoning process. other ground-truths forces the model to generate excessively long, exhaustive captions to artificially m… view at source ↗
Figure 5
Figure 5. Figure 5: Exemplars from the TimeThink-RFT-20K dataset. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 37 linked inside Pith

  1. [1]

    In: Proceedings of the IEEE international conference on computer vision

    Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with natural language. In: Proceedings of the IEEE international conference on computer vision. pp. 5803–5812 (2017)

    2017

  2. [2]

    arXiv preprint arXiv:2404.03413 (2024)

    Ataallah, K., Shen, X., Abdelrahman, E., Sleiman, E., Zhu, D., Ding, J., Elhoseiny, M.: Minigpt4-video: Advancing multimodal llms for video understanding with in- terleaved visual-textual tokens. arXiv preprint arXiv:2404.03413 (2024)

    Pith/arXiv arXiv 2024

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    com / Breakthrough / PySceneDetect, software available from https://www.scenedetect.com

    Castellano, B.: PySceneDetect: Video Cut Detection and Analysis Tool (2026), https : / / github . com / Breakthrough / PySceneDetect, software available from https://www.scenedetect.com

    2026

  6. [6]

    arXiv preprint arXiv:2412.12075 (2024)

    Chen, G., Liu, Y., Huang, Y., He, Y., Pei, B., Xu, J., Wang, Y., Lu, T., Wang, L.: Cg-bench: Clue-grounded question answering benchmark for long video under- standing. arXiv preprint arXiv:2412.12075 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    arXiv preprint arXiv:2406.04325 (2024)

    Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z., et al.: Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    arXiv preprint arXiv:2411.18211 (2024)

    Chen, S., Lan, X., Yuan, Y., Jie, Z., Ma, L.: Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv:2411.18211 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Advances in Neural Information Processing Systems36, 72842–72866 (2023)

    Chen,S.,Li,H.,Wang,Q.,Zhao,Z.,Sun,M.,Zhu,X.,Liu,J.:Vast:Avision-audio- subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems36, 72842–72866 (2023)

    2023

  10. [10]

    arXiv preprint arXiv:2412.05271 (2024)

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far are we to gpt-4v? closing the gap to commercial multimodal modelswith open-source suites.Science China InformationSciences67(12),220101 (2024)

    2024

  12. [12]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning (2023)

    2023

  14. [14]

    arXiv preprint arXiv:2408.14023 (2024)

    Fei, J., Li, D., Deng, Z., Wang, Z., Liu, G., Wang, H.: Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    arXiv preprint arXiv:2503.21776 (2025) 16 H

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025) 16 H. Li et al

    Pith/arXiv arXiv 2025

  16. [16]

    arXiv preprint arXiv:2405.21075 (2024)

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075 (2024)

    Pith/arXiv arXiv 2024

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: An empirical study of end-to-end video-language transformers with masked visual modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22898–22909 (2023)

    2023

  18. [18]

    In: Proceedings of the IEEE international conference on computer vision

    Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: Temporal activity localization via language query. In: Proceedings of the IEEE international conference on computer vision. pp. 5267–5275 (2017)

    2017

  19. [19]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    arXiv preprint arXiv:2501.13826 (2025)

    Hu, K., Wu, P., Pu, F., Xiao, W., Zhang, Y., Yue, X., Li, B., Liu, Z.: Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    arXiv preprint arXiv:2404.06395 (2024)

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al.: Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, B., Wang, X., Chen, H., Song, Z., Zhu, W.: Vtimellm: Empower llm to grasp video moments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14271–14280 (2024)

    2024

  23. [23]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  24. [24]

    arXiv preprint arXiv:2412.16720 (2024)

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    In: Findings of the Association for Computational Lin- guistics: ACL 2025

    Kahatapitiya, K., Ranasinghe, K., Park, J., Ryoo, M.S.: Language repository for long video understanding. In: Findings of the Association for Computational Lin- guistics: ACL 2025. pp. 5627–5646 (2025)

    2025

  26. [26]

    In: Proceedings of the IEEE international conference on computer vision

    Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Carlos Niebles, J.: Dense-captioning events in videos. In: Proceedings of the IEEE international conference on computer vision. pp. 706–715 (2017)

    2017

  27. [27]

    Li, B., Zhang, P., Zhang, K., Pu, F., Du, X., Dong, Y., Liu, H., Zhang, Y., Zhang, G., Li, C., Liu, Z.: Lmms-eval: Accelerating the development of large multimoal models (March 2024),https://github.com/EvolvingLMMs-Lab/lmms-eval

    2024

  28. [28]

    arXiv preprint arXiv:2503.18422 (2025)

    Li, H., Zhang, Y., Guo, L., Yue, X., Liu, J.: Breaking the encoder barrier for seamless video-language understanding. arXiv preprint arXiv:2503.18422 (2025)

    arXiv 2025

  29. [29]

    arXiv preprint arXiv:2311.17005 (2023)

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005 (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024)

    2024

  31. [31]

    arXiv preprint arXiv:2504.06958 (2025) TimeThink: Reasoning with Time for Video LLMs 17

    Li,X.,Yan,Z.,Meng,D.,Dong,L.,Zeng,X.,He,Y.,Wang,Y.,Qiao,Y.,Wang,Y., Wang, L.: Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958 (2025) TimeThink: Reasoning with Time for Video LLMs 17

    Pith/arXiv arXiv 2025

  32. [32]

    Li, Y., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large lan- guagemodels.In:EuropeanConferenceonComputerVision.pp.323–340.Springer (2025)

    2025

  33. [33]

    arXiv preprint arXiv:2509.18056 (2025)

    Li, Y., Cheng, J., Jia, S., Kuang, H., Jiao, S., Hou, Q., Cheng, M.M.: Tempsamp-r1: Effective temporal sampling with reinforcement fine-tuning for video llms. arXiv preprint arXiv:2509.18056 (2025)

    arXiv 2025

  34. [34]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learn- ing united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre- training for visual language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26689–26699 (2024)

    2024

  36. [36]

    Advances in neural information processing systems36(2024)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36(2024)

    2024

  37. [37]

    arXiv preprint arXiv:2503.13444 (2025)

    Liu, Y., Lin, K.Q., Chen, C.W., Shou, M.Z.: Videomind: A chain-of-lora agent for long video reasoning. arXiv preprint arXiv:2503.13444 (2025)

    arXiv 2025

  38. [38]

    arXiv preprint arXiv:2306.05424 (2023)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards de- tailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    2023

  40. [40]

    ArXivabs/2303.08774(2023)

    OpenAI: Gpt-4 technical report. ArXivabs/2303.08774(2023)

    Pith/arXiv arXiv 2023

  41. [41]

    arXiv preprint arXiv:2504.01407 (2025)

    Pan, J., Zhang, R., Wan, X., Zhang, Y., Lu, M., She, Q.: Timesearch: Hierarchical video search with spotlight and reflection for human-like long video understanding. arXiv preprint arXiv:2504.01407 (2025)

    arXiv 2025

  42. [42]

    Advances in Neural Information Processing Systems37, 119336–119360 (2024)

    Qian, R.,Dong,X.,Zhang,P., Zang,Y.,Ding,S.,Lin,D., Wang,J.:Streaminglong video understanding with large language models. Advances in Neural Information Processing Systems37, 119336–119360 (2024)

    2024

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Qu,M.,Chen,X.,Liu,W.,Li,A.,Zhao,Y.:Chatvtg:Videotemporalgroundingvia chat with video dialogue large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1847–1856 (2024)

    2024

  44. [44]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  45. [45]

    arXiv preprint arXiv:2403.05530 (2024)

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    Pith/arXiv arXiv 2024

  46. [46]

    arXiv preprint arXiv:2410.03290 (2024)

    Wang,H.,Xu,Z.,Cheng,Y.,Diao,S.,Zhou,Y.,Cao,Y.,Wang,Q.,Ge,W.,Huang, L.: Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03290 (2024)

    Pith/arXiv arXiv 2024

  47. [47]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., Lin, J.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  48. [48]

    arXiv preprint arXiv:2505.12434 (2025)

    Wang, Q., Yu, Y., Yuan, Y., Mao, R., Zhou, T.: Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning. arXiv preprint arXiv:2505.12434 (2025)

    arXiv 2025

  49. [49]

    arXiv preprint arXiv:2503.13377 (2025) 18 H

    Wang, Y., Wang, Z., Xu, B., Du, Y., Lin, K., Xiao, Z., Yue, Z., Ju, J., Zhang, L., Yang, D., et al.: Time-r1: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377 (2025) 18 H. Li et al

    Pith/arXiv arXiv 2025

  50. [50]

    arXiv e-prints pp

    Wang, Y., Xu, B., Yue, Z., Xiao, Z., Wang, Z., Zhang, L., Yang, D., Wang, W., Jin, Q.: Timezero: Temporal video grounding with reasoning-guided lvlm. arXiv e-prints pp. arXiv–2503 (2025)

    2025

  51. [51]

    arXiv preprint arXiv:2403.10228 (2024)

    Wang, Y., Meng, X., Liang, J., Wang, Y., Liu, Q., Zhao, D.: Hawkeye: Train- ing video-text llms for grounding text in videos. arXiv preprint arXiv:2403.10228 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Pro- cessing Systems37, 28828–28857 (2024)

    2024

  53. [53]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition

    Xiao, J., Yao, A., Li, Y., Chua, T.S.: Can i trust your answer? visually grounded video question answering. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 13204–13214 (2024)

    2024

  54. [54]

    In: ACM Multimedia (2017)

    Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., Zhuang, Y.: Video question answering via gradually refined attention over appearance and motion. In: ACM Multimedia (2017)

    2017

  55. [55]

    arXiv preprint arXiv:2404.16994 (2024)

    Xu, L., Zhao, Y., Zhou, D., Lin, Z., Ng, S.K., Feng, J.: Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994 (2024)

    Pith/arXiv arXiv 2024

  56. [56]

    5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception

    Yan, Z., Li, X., He, Y., Yue, Z., Zeng, X., Wang, Y., Qiao, Y., Wang, L., Wang, Y.: Videochat-r1. 5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception. arXiv preprint arXiv:2509.21100 (2025)

    arXiv 2025

  57. [57]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yan, Z., Li, Z., He, Y., Wang, C., Li, K., Li, X., Zeng, X., Wang, Z., Wang, Y., Qiao,Y.,etal.:Taskpreferenceoptimization:Improvingmultimodallargelanguage models with vision task alignment. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29880–29892 (2025)

    2025

  58. [58]

    arXiv preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Advances in Neural Information Processing Systems35, 124–141 (2022)

    Yang, A., Miech, A., Sivic, J., Laptev, I., Schmid, C.: Zero-shot video question an- swering via frozen bidirectional language models. Advances in Neural Information Processing Systems35, 124–141 (2022)

    2022

  60. [60]

    In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference

    Yang, J., Yang, S., Gupta, A.W., Han, R., Fei-Fei, L., Xie, S.: Thinking in space: How multimodal large language models see, remember, and recall spaces. In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference. pp. 10632– 10643 (2025)

    2025

  61. [61]

    arXiv preprint arXiv:2305.06988 (2023)

    Yu, S., Cho, J., Yadav, P., Bansal, M.: Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988 (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    In: Pro- ceedings of the AAAI Conference on Artificial Intelligence

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 9127–9134 (2019)

    2019

  63. [63]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Zhang, C., Lu, T., Islam, M.M., Wang, Z., Yu, S., Bansal, M., Bertasius, G.: A simple llm framework for long-range video question-answering. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 21715–21737 (2024)

    2024

  64. [64]

    Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J.A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., Liu, Z.: Lmms-eval: Reality check on the evaluation of large multimodal models (2024),https://arxiv.org/abs/2407.12772

    Pith/arXiv arXiv 2024

  65. [65]

    arXiv preprint arXiv:2406.16852 (2024) TimeThink: Reasoning with Time for Video LLMs 19

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., Liu, Z.: Long context transfer from language to vision. arXiv preprint arXiv:2406.16852 (2024) TimeThink: Reasoning with Time for Video LLMs 19

    Pith/arXiv arXiv 2024

  66. [66]

    arXiv preprint arXiv:2406.09412 (2024)

    Zhang, Y., Li, H., Liu, J., Yue, X.: Explore the limits of omni-modal pretraining at scale. arXiv preprint arXiv:2406.09412 (2024)

    Pith/arXiv arXiv 2024

  67. [67]

    arXiv preprint arXiv:2410.02713 (2024)

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024)

    Pith/arXiv arXiv 2024

  68. [68]

    arXiv preprint arXiv:2503.05379 (2025)

    Zhao, J., Wei, X., Bo, L.: R1-omni: Explainable omni-multimodal emotion recog- nition with reinforcement learning. arXiv preprint arXiv:2503.05379 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al.: Mmvu: Measuring expert-level multi-discipline video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8475–8489 (2025)

    2025

  70. [70]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., et al.: Swift: a scalable lightweight infrastructure for fine-tuning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 29733– 29735 (2025)

    2025

  71. [71]

    arXiv preprint arXiv:2406.09367 (2024)

    Zhao, Z., Lu, H., Huo, Y., Du, Y., Yue, T., Guo, L., Wang, B., Chen, W., Liu, J.: Needle in a video haystack: A scalable synthetic evaluator for video mllms. arXiv preprint arXiv:2406.09367 (2024)

    Pith/arXiv arXiv 2024

  72. [72]

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y., Zhang, B., Huang, T., Liu, Z.: Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264 (2024) TimeThink: Reasoning with Time for Video LLMs 1 TimeThink: Reasoning with Time for Video LLMs Supplementary Material 8 Additional Ablation Studies...

    Pith/arXiv arXiv 2024

  73. [73]

    Understand the Target QA: Determine the exact visual/semantic evidence re- quired

  74. [74]

    Use the Full Video Context for background, but base your scorestrictlyon the Candidate Video Clip

    Evaluate the Candidate Clip: Determine if the specific clip contains this evi- dence. Use the Full Video Context for background, but base your scorestrictlyon the Candidate Video Clip

  75. [75]

    Clear, unambiguous, and complete evidence

    Scoring Criteria (0 to 10 Scale): -10: Perfect Relevance. Clear, unambiguous, and complete evidence. -7-9: High Relevance. Strong evidence, might lack minor context. -4-6: Partial Relevance. Partial, ambiguous, or indirect evidence. -1-3: Low Relevance. Mostly irrelevant, tangential elements present. -0: No Relevance. Completely irrelevant

  76. [76]

    initial setting

    Provide concise REASONS (1-3 sentences) explaining the score before out- putting the SCORE. User: INPUT DATA: Target QUESTION: {question} Target ANSWER: {answer} Full Video Context (For background understanding only): {full_video_content} Candidate Video Clip ([{clip_start_time} - {clip_end_time}] seconds): {clip_content} STRICT OUTPUT FORMAT: REASONS: <Y...