Pith. sign in

REVIEW 3 major objections 4 minor 43 references

The paper argues that making a video captioner first emit a structured spatiotemporal perception trace—object trajectories with bounding boxes and events with time spans—and only then write the caption produces more detailed, better-grounde

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:56 UTC pith:Q4ZOZNPU

load-bearing objection A coherent, well-audited pipeline for perception-aware video captioning with consistent gains, but the central mechanism claim rests on a missing inference-time control. the 3 major comments →

arxiv 2607.20389 v1 pith:Q4ZOZNPU submitted 2026-07-22 cs.CV

PercepCap: Video Captioner with Structured Spatio-Temporal Perception

Yifan Xu , Zihao Wang , Zhixiao Wang , Jiaming Zhang , Yichun Yang , Desen Meng , Yuanxing Zhang , Pengfei Wan
show 1 more author
Limin Wang
This is my paper
classification cs.CV
keywords video captioningspatio-temporal perceptionmultimodal large language modelsperception tracereinforcement learningobject trackingtemporal groundingdetailed video description
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PercepCap tries to establish that detailed video captioning improves when the model is forced to expose its perceptual evidence before writing the caption. Instead of mapping the video directly to text, it generates a structured spatiotemporal perception trace—objects identified and tracked across frames with bounding boxes, and events localized to start/end times—and only then produces the final description conditioned on that trace. The paper trains this two-stage behavior with a supervised fine-tuning step followed by reinforcement learning whose rewards separately score object localization, event timing, event semantics, and caption coverage. Across several captioning and caption-to-QA benchmarks, it reports consistent gains over the caption-only base model, and its ablations attribute the gain to the explicit perception interface rather than to extra caption supervision.

Core claim

The central claim is that the perceive–describe generation chain itself is what improves captioning, not just better caption data or more RL. By making object trajectories and temporally grounded events a required intermediate output, PercepCap turns caption generation into evidence-grounded generation: the final caption is a verbalization of a trace the model has already committed to. The training data for this chain is built by caption-anchored construction—first writing the caption, extracting the objects and events it mentions, then grounding those mentions back in the video with boxes and timestamps—so the trace and caption are guaranteed to refer to the same content. PercepCap then opt

What carries the argument

The load-bearing object is the structured spatiotemporal perception trace, a machine-readable JSON block the model must emit before the caption. It has two parts: objects, each with a temporally consistent id, a semantic label, and a localization list of timestamped 2D bounding boxes; and events, each with start/end timestamps and a natural-language description whose subjects are meant to match the tracked objects. The trace is the interface that makes the paper's claims work: it converts a single undifferentiated captioning task into two supervised stages, gives RL a place to attach separate perception- and caption-level rewards, and provides a human-auditable record of what the model perce

Load-bearing premise

The perception ground truth used for training comes from an automatic model that first writes the caption and then re-grounds that caption's own mentions in the video, with no human verification; if those automatically produced boxes and timestamps are wrong or merely paraphrase the caption, the perceived gain could come from conditioning on an external model's opinion rather than from better perception.

What would settle it

At inference, randomize the order of object ids and events in the perception trace before the caption is generated; if final caption quality does not drop relative to the intact trace, the model is not actually using the trace. A complementary check: replace the automatic trace with human-verified boxes and timestamps and see whether caption scores improve—if they do not, the paper's training signal, not its architecture, is the weak point.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Caption-only datasets can be repurposed as perception-supervision data: write the caption, extract its objects and events, ground them with boxes and timestamps, and use the result for supervised and reinforcement training.
  • Caption errors become diagnosable: a failure can be traced to whether the model missed an object, placed an event at the wrong time, or verbalized correct evidence poorly.
  • Reinforcement learning gets finer credit assignment: separate rewards for spatial, temporal, semantic, and caption components can push up each dimension instead of optimizing only final text.
  • Captions produced this way retain more of the information needed for downstream tasks, since caption-to-QA accuracy also improves.
  • Because the trace is explicit, the pipeline is auditable: intermediate perception can be inspected or corrected before the caption is trusted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A decisive test the paper does not run: swap in independently human-verified object tracks and event boundaries at inference. If captions do not improve beyond the caption-only baseline, the benefit would be more about conditioning on extra text than about genuine perception.
  • The same caption-anchored data construction could be applied to domain-specific captions (sports, egocentric video, surveillance) to bootstrap perception traces for new settings without manual annotation—though the paper does not demonstrate this.
  • If the trace is essentially a structured paraphrase of the caption, then the method's improvement may partly stem from inference-time chain-of-thought decoding rather than new perceptual capability; varying trace granularity would separate these effects.
  • The machine-readable trace (boxes plus timestamps) suggests a natural extension to retrieval or grounded QA, where the evidence can be consumed directly rather than re-extracted from text.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents PercepCap, a video captioning framework that inserts a structured spatio-temporal perception trace (object trajectories with bounding boxes and temporally grounded events) before final caption generation. Training uses two stages: Perceive-then-Describe SFT (PD-SFT) and Perception-Grounded RL (PG-RL) with a composite reward over object localization, event timestamps, event semantics, format, and caption coverage. To obtain paired trace/caption supervision, the authors propose Caption-Anchored Perception Data Construction, in which Gemini 3 Flash generates a caption-only anchor, the objects/events in that caption are extracted, and Gemini 3.1 Pro grounds them back into the video with boxes and timestamps. Experiments on DREAM-1K, CaReBench, ShortVidBench, MotionBench, and VidCapBench-AE show consistent improvements over the Qwen3-VL-8B caption-only baseline; ablations isolate several training choices.

Significance. If the mechanism holds, PercepCap is a useful step toward auditable, evidence-grounded video captioning and provides a practical recipe for converting caption-only MLLMs into perceive-describe chains. The paper's strengths are its consistent multi-benchmark gains, architecture-agnostic training recipe, public backbone, and an unusually candid limitations section. The main conceptual claim — that the explicit perception trace itself, not merely additional training on a serialized target or an external model's opinion, is what drives improvement — is currently not isolated by the experiments; the missing control is inexpensive to run and would substantially raise confidence.

major comments (3)
  1. [§5.4, Table 3; §G] The central claim that the perception trace improves captioning is not tested at inference. #5 vs #3 and #6 vs #4 differ in both the training objective and the inference prompt, so the gains cannot be attributed to the trace itself. #2 conditions on an external Gemini trace, not on the trained model's self-generated trace. The necessary control — running the PD-SFT or full PercepCap checkpoint with the caption-only prompt (Figure 10), or otherwise suppressing trace generation at inference — is absent. Without that cell, the +0.9 F1 from #5 to #7 and the gap vs #3 could come from training on the serialized caption target rather than from the inference-time perceive-describe chain. This is directly testable and should be added.
  2. [§4, §5.5, Table 4] The perception supervision is circular with respect to the caption: Caption-Anchored construction first generates the anchor caption, extracts its objects/events, and then grounds them with the same external model family (Gemini). The paper acknowledges error propagation in §5.5, but the claimed mechanism — perception evidence improves captioning — is not validated by independent perception ground truth. Table 4's perception probe prompts both variants to produce a trace under the same structured prompt, so it measures post-hoc perception ability, not the contribution of the trace inside PercepCap's actual inference chain. I recommend adding a human-verified or independently annotated perception-evaluation subset, and/or a manipulation experiment in which trace quality is perturbed and caption quality is measured.
  3. [§5.3 and Table 1] Tables 1 and 2 report single-point comparisons without variance or significance testing. Given that several margins over the strongest open-source baselines are small (e.g., ShortVidBench 40.4 vs 40.3 for Gemini 3 Flash and 39.7 for GLM; VidCapBench-AE Overall 19.0 vs 17.8), reporting standard errors over multiple evaluation runs or per-benchmark bootstrap intervals would clarify whether the improvements are robust rather than noise-level. This is not a blocker but is load-bearing for the claim of 'leading caption quality.'
minor comments (4)
  1. [Throughout] Typos and formatting issues: 'Univerisity' in the affiliation, 'serveral' in §1, 'model, outputs' in §4 Step 2, 'eρ i' rendering in Eq. (3), and inconsistent F1/R/P ordering in Table 1 headers (the text defines AutoDQ as F1/Precision/Recall, but the headers list F1/R/P).
  2. [Figures 7 and 8] The figure captions appear swapped: Figure 7 shows the object-extraction prompt but is captioned 'Prompt Template for Event Extraction of Training Data', while Figure 8 shows the event-extraction prompt but is captioned for object extraction. Please correct.
  3. [Table 3] Abbreviations such as 'R_cap only RL' are not defined in the table caption; define them in the text. Also specify whether #2 uses the same 32-frame sampling as the trained variants, and whether the external Gemini trace is produced with the same perception prompt as the model's own trace.
  4. [§5.1 and §C] The reward coefficients in Eq. (5) are fixed to α=β=δ=0.5 and γ=λ=1.0, but no sensitivity analysis is reported. Even a small grid would strengthen the claim that the composite reward is robust and that the perception rewards matter beyond the caption reward.

Circularity Check

0 steps flagged

No significant circularity: the central caption-quality claim is evaluated on external benchmarks, and the perception trace is an intermediate output, not a fitted input.

full rationale

The paper's derivation chain is empirical rather than formal. Training uses Caption-Anchored Perception Data Construction, where a Gemini-generated caption is used to extract objects/events and ground them in the video, making the perception trace caption-aligned by construction (Section 4). The paper itself flags the corresponding limitation in Section 5.5: 'its supervision depends on annotator-generated perception data rather than manually verified object tracks and event boundaries, so errors in the constructed traces can propagate to SFT and RL references.' This is a data-dependence/quality limitation, not a circular reduction. At inference the model must generate the trace from the video itself, and the final caption (not the trace) is scored against external benchmarks (DREAM-1K, CaReBench, ShortVidBench, MotionBench, VidCapBench-AE) with human/LLM references. The ablations in Table 3 compare caption-only vs perception-aware training on matched captions (#3 vs #5, #4 vs #6), providing an empirical test of the trace's contribution. The missing control (trained model with caption-only prompt) is an experimental gap, not circularity. The only self-citations (CaReBench for LLM-judge validity, TimeChat-Captioner for SODA_m protocol) are methodological and not load-bearing for the central result. No equation reduces to its own input, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. Its load-bearing assumptions are about the validity of LLM-generated perception supervision, the unbiasedness of LLM judges, and the usefulness of the fixed trace schema.

free parameters (1)
  • Reward coefficients α, β, δ, γ, λ = α=β=δ=0.5, γ=λ=1.0
    Set by hand to balance perception and caption rewards (Eqn. 5); no sweep or sensitivity analysis is reported, yet the RL objective's behavior depends on these values.
axioms (3)
  • domain assumption Objects and events extracted from a caption-only description and grounded back in the video by Gemini constitute correct perception ground truth.
    Section 4 and Appendix F: the entire training and reward signal rests on this pipeline; the paper's own §5.5 acknowledges errors can propagate.
  • domain assumption LLM judges (GPT-4.1, Gemini 3.1 Pro, GPT-4o) provide valid evaluations of caption quality that are not biased toward the LLM-generated training data style.
    Section 5.2: all benchmarks use LLM judges; if judge preferences correlate with Gemini/GPT output style, the reported gains could reflect style matching rather than perceptual truth.
  • domain assumption A structured JSON perception trace with fixed schema is a sufficient and useful intermediate representation for detailed video captioning.
    Section 3.1: the method reformulates captioning as generating this trace before the caption; the paper acknowledges the fixed schema may be inflexible (Limitations §5.5).

pith-pipeline@v1.3.0-alltime-deepseek · 15357 in / 10583 out tokens · 83397 ms · 2026-08-01T09:56:55.059403+00:00 · methodology

0 comments
read the original abstract

Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.

Figures

Figures reproduced from arXiv: 2607.20389 by Desen Meng, Jiaming Zhang, Limin Wang, Pengfei Wan, Yichun Yang, Yifan Xu, Yuanxing Zhang, Zhixiao Wang, Zihao Wang.

Figure 1
Figure 1. Figure 1: Overview of PercepCap’s perceive–describe chain. The model first generates a structured spatio-temporal perception trace and then produces the final caption conditioned on this evidence. Together, these studies suggest that RL-based post-training is a promising way to improve video perception, reasoning, and caption generation in MLLMs. PercepCap builds on this direction, but shifts the focus from rewardin… view at source ↗
Figure 2
Figure 2. Figure 2: Example and construction flow for Caption-Anchored Perception Data Construction. A [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison between PercepCap and a direct single-step captioning baseline. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Additional qualitative comparison between PercepCap and a direct single-step captioning [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Constructed training example from Caption-Anchored Perception Data Construction. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Constructed training example from Caption-Anchored Perception Data Construction. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for extracting object labels from the caption during data construction. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt for extracting event descriptions from the caption during data construction. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt for completing video-grounded perception fields during data construction. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt for caption-only inference. You are given a full frame-by-frame sequence of a video. Each frame is preceded by its timestamp in the form <xx seconds>. Your task is to observe the video, produce a structured perception trace, and then describe the video in detail. First, observe the video according to the following two tasks. Task A: Object Localization - For each object in the video, provide its bo… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for perception-aware PercepCap inference with structured output. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 19 linked inside Pith

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.CoRR, abs/2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.CoRR, abs/2308.12966, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    Pith/arXiv arXiv 2025

  3. [3]

    Activitynet: A large- scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015

    2015

  4. [4]

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq-Neng Hwang, Saining Xie, and Christopher D. Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark.CoRR, abs/2410.03051, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    VidCapBench: a comprehensive benchmark of video captioning for controllable text-to-video generation.CoRR, abs/2502.12782, 2025

    Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chengru Song, Qiang Liu, Di Zhang, and Tieniu Tan. VidCapBench: a comprehensive benchmark of video captioning for controllable text-to-video generation.CoRR, abs/2502.12782, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Vidbridge-r1: Bridging QA and captioning for RL-based video understanding models with intermediate proxy tasks

    Xinlong Chen, Yuanxing Zhang, Yushuo Guan, Weihong Lin, Zekun Moore Wang, Bohan Zeng, Yang Shi, Sihan Yang, Qiang Liu, Pengfei Wan, and Liang Wang. Vidbridge-r1: Bridging QA and captioning for RL-based video understanding models with intermediate proxy tasks. InThe F ourteenth International Conference on Learning Representations, 2026

    2026

  7. [7]

    How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    ARC-Hunyuan-Video-7B: structured video comprehension of real-world shorts.CoRR, abs/2507.20939, 2025

    Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, Jinwen Luo, Weibo Gu, Zexuan Li, Xiaojing Zhang, Yangyu Tao, Han Hu, Di Wang, and Ying Shan. ARC-Hunyuan-Video-7B: structured video comprehension of real-world shorts.CoRR, abs/2507.20939, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Gemini 3 Flash: frontier intelligence built for speed

    Google. Gemini 3 Flash: frontier intelligence built for speed. Google Blog, 2025

    2025

  11. [11]

    MotionBench: benchmarking and improving fine-grained video motion understanding for vision language models.CoRR, abs/2501.02955, 2025

    Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. MotionBench: benchmarking and improving fine-grained video motion understanding for vision language models.CoRR, abs/2501.02955, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    VIDEOP2R: video understanding from perception to reasoning.CoRR, abs/2511.11113, 2025

    Yifan Jiang, Yueying Wang, Rui Zhao, Toufiq Parag, Zhimin Chen, Zhenyu Liao, and Jayakrishnan Unnikrishnan. VIDEOP2R: video understanding from perception to reasoning.CoRR, abs/2511.11113, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Kimi-VL technical report.CoRR, abs/2504.07491, 2025

    Kimi Team. Kimi-VL technical report.CoRR, abs/2504.07491, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models.CoRR, abs/2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models.CoRR, abs/2407.07895, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. CoRR, abs/2504.06958, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 5971–5984. Association for Computational Linguistics, 2024

    2024

  17. [17]

    MiMo-VL technical report, 2025

    LLM-Core-Team Xiaomi. MiMo-VL technical report, 2025

    2025

  18. [18]

    Videocap-r1: Enhancing mllms for video captioning via structured thinking.CoRR, abs/2506.01725, 2025

    Desen Meng, Rui Huang, Zhilin Dai, Xinhao Li, Yifan Xu, Jun Zhang, Zhenpeng Huang, Meng Zhang, Lingshu Zhang, Yi Liu, and Limin Wang. Videocap-r1: Enhancing mllms for video captioning via structured thinking.CoRR, abs/2506.01725, 2025. 11

    Pith/arXiv arXiv 2025

  19. [19]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human fee...

    2022

  20. [20]

    Qwen3.5: Towards native multimodal agents, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, 2026

    2026

  21. [21]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InCVPR, pages 14313–14323. IEEE, 2024

    2024

  22. [22]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  23. [23]

    Gemini 3.1 pro: A smarter model for your most complex tasks, 2026

    The Gemini Team. Gemini 3.1 pro: A smarter model for your most complex tasks, 2026. Accessed: 2026-05-07

    2026

  24. [24]

    Reft: Reasoning with reinforced fine-tuning

    Luong Quoc Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. InACL (1), pages 7601–7614. Association for Computational Linguistics, 2024

    2024

  25. [25]

    GLM-4.5V and GLM-4.1V-Thinking: towards versatile multimodal reasoning with scalable reinforcement learning.CoRR, abs/2507.01006, 2026

    V Team. GLM-4.5V and GLM-4.1V-Thinking: towards versatile multimodal reasoning with scalable reinforcement learning.CoRR, abs/2507.01006, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Tarsier: Recipes for training and evaluating large video description models.CoRR, abs/2407.00634, 2024

    Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models.CoRR, abs/2407.00634, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.CoRR, abs/2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.CoRR, abs/2409.12191, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas S. Huang. YouTube-VOS: a large-scale video object segmentation benchmark.CoRR, abs/1809.03327, 2018

    Pith/arXiv arXiv 2018

  29. [29]

    CaReBench: A fine-grained benchmark for video captioning and retrieval.CoRR, abs/2501.00513, 2025

    Yifan Xu, Xinhao Li, Yichun Yang, Desen Meng, Rui Huang, and Limin Wang. CaReBench: A fine-grained benchmark for video captioning and retrieval.CoRR, abs/2501.00513, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Videochat-r1.5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception

    Ziang Yan, Xinhao Li, Yinan He, Zhengrong Yue, Xiangyu Zeng, Yali Wang, Yu Qiao, Limin Wang, and Yi Wang. Videochat-r1.5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception. CoRR, abs/2509.21100, 2025

    arXiv 2025

  31. [31]

    Timechat- captioner: Scripting multi-scene videos with time-aware and structural audio-visual captions.CoRR, abs/2602.08711, 2026

    Linli Yao, Yuancheng Wei, Yaojie Zhang, Lei Li, Xinlong Chen, Feifan Song, Ziyue Wang, Kun Ouyang, Yuanxin Liu, Lingpeng Kong, Qi Liu, Pengfei Wan, Kun Gai, Yuanxing Zhang, and Xu Sun. Timechat- captioner: Scripting multi-scene videos with time-aware and structural audio-visual captions.CoRR, abs/2602.08711, 2026

    Pith/arXiv arXiv 2026

  32. [32]

    Timelens: Rethinking video temporal grounding with multimodal llms.CoRR, abs/2512.14698, 2025

    Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms.CoRR, abs/2512.14698, 2025

    arXiv 2025

  33. [33]

    perception

    Chunlin Zhong, Qiuxia Hou, Zhangjun Zhou, Yanhao Zhang, Shuang Hao, Haonan Lu, He Tang, and Xiang Bai. OwlCap: harmonizing motion-detail for video captioning via HMD-270K and caption set equivalence reward. InAAAI, pages 13503–13511, 2026. 12 A Supplementary Overview This appendix collects implementation and evaluation details that support the main paper ...

    2026

  34. [34]

    Caption Anchor Generation.The annotator model first receives the video in a caption-only setting and generates the final captionY ⋆ cap

  35. [35]

    These extracted textual contents define which entities and events must appear in the perception trace

    Object-Event Content Extraction.The annotator model extracts object labels and event descriptions from Y ⋆ cap. These extracted textual contents define which entities and events must appear in the perception trace

  36. [36]

    14 Figure 5: Constructed training example from Caption-Anchored Perception Data Construction

    Video-Grounded Perception Completion.Conditioned on the video and extracted contents, the annotator model completes the missing perception fields by assigning timestamped boxes to object labels and start/end times to event descriptions. 14 Figure 5: Constructed training example from Caption-Anchored Perception Data Construction

  37. [37]

    objects". - The value of

    Structured Data Assembly.The completed objects and events are assembled with the original caption into a structured data instance Y ⋆ = [Y ⋆ perc, Y⋆ cap] for PD-SFT and as reference evidence for PG-RL. This procedure enforces object-event consistency by construction, because the perception trace is grounded from the same caption content that will be used...

  38. [38]

    - The same real-world instance MUST keep the same id across frames

    id: an integer that is unique across the entire video and temporally consistent across frames. - The same real-world instance MUST keep the same id across frames. - If an instance disappears and reappears, reuse the same id. - Create a new id only for a different real-world instance

  39. [39]

    label: a fine-grained semantic label to the best of your certainty

  40. [40]

    - bbox_2d: [xmin, ymin, xmax, ymax] in relative pixel coordinates, range [0, 1000]

    localization: a list of dictionaries, one for each sampled frame: - timestamp: the frame timestamp, formatted as <xx seconds>. - bbox_2d: [xmin, ymin, xmax, ymax] in relative pixel coordinates, range [0, 1000]. - If the object is not present in a frame, output bbox_2d as []. Task B: Event Detection and Temporal Grounding - Output ALL distinguishable event...

  41. [41]

    event: a detailed natural-language description

  42. [42]

    start_time: the event start timestamp, formatted as <xx seconds>

  43. [43]

    objects":[{

    end_time: the event end timestamp, formatted as <yy seconds>. - Events should cover salient action changes, interactions, state changes, and scene transitions. - Avoid redundant events that add no new information. Output Requirements - Put the structured perception trace inside <perception></perception> tags. - The content inside <perception></perception>...