Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IntentVCNet claims that giving a vision-language model per-frame box coordinates, a red-box overlay, and region-injected features makes it generate intention-focused video captions, reaching CIDEr 225.19 and second place in the IntentVC…

desk verdict A useful challenge entry with an honest ablation and a genuinely informative adapter-placement study, but the headline CIDEr 225.19 is not tied to a described inference recipe, so the SOTA claim needs a full ensemble specification before it can be trusted. read the letter →

arxiv 2507.18531 v1 pith:NP3EVX7Y submitted 2025-07-24 cs.CV

Tianheng Qiu , Jingchun Gao , Jingyu Li , Huiyi Leong , Xuan Huang , Xi Wang , Xiaocheng Zhang , Kele Xu
show 1 more author
Lan Zhang
This is my paper
classification cs.CV
keywords intention-orientedvideocaptioningcontrollablelargevision-languagemodelspromptcombinationboxadaptervisualgroundingensemblelearningIntentVC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the gap between spatial grounding and temporal video understanding in large vision-language models can be closed for intention-oriented captioning by treating the user's target object as a per-frame bounding box and making that box visible to the model in three redundant ways: as normalized coordinates in the instruction, as a red box drawn on the video frames, and as a region feature injected into the visual encoder by a box adapter. On the IntentVC benchmark, the resulting system reaches a CIDEr of 225.19 on the public test set, ahead of four fine-tuned open LVLMs, and ranked second in the IntentVC Challenge. The claim matters because it suggests box-level intent can be a practical control signal for personalized video captioning without changing the base models or training from scratch.

What carries the argument

The key mechanism is the box adapter, a parameter-efficient module inserted after the last five layers of the vision transformer's ViT. It first extracts region-of-interest features from the provided box with RoI Align, then runs cross-attention where the full frame feature map is the query and the region features are the keys and values, with a zero-initialized convolution to keep early training stable. This turns the global visual context into a region-enhanced feature map. Around it, the prompt combination gives the language model normalized box coordinates in text and a red box on every frame, and the final ensemble uses text similarity to pick the caption that multiple heterogeneous models agree on.

What would settle it

Take the IntentVC public test set and replace the ground-truth boxes with detector boxes or boxes corrupted by random translation and scale jitter, keeping the same training and inference script; if CIDEr stays near 225.19, box precision is not critical, and if it falls sharply, the method's controllability is confined to box-supervised benchmarks. A complementary check is to supply only the object category in the instruction, with no coordinates or boxes, and observe whether the generated caption still identifies and tracks the intended object.

Watch

Extended reading notes

Core claim

The central claim is that combining textual coordinate prompts, red-box visual prompts, and region-enhanced visual features lets an open LVLM track a user-specified object through a video and describe that object's actions while retaining scene context. The paper reports that textual prompts alone raise CIDEr from 196.2 to 211.45, the box adapter raises the textual-prompt model to 223.01, and voting over two heterogeneous models yields 225.19, outperforming the strongest fine-tuned baseline by 37.71 CIDEr points. The intended reading is that both prompt-side and model-side spatial conditioning are needed, and that the two prompt modalities work best when kept in separate models and combined by consensus rather than merged into one prompt.

Load-bearing premise

The load-bearing premise is that the user's intent is provided as an accurate per-frame bounding box at test time; if boxes are missing, noisy, or replaced by a mere object category, the coordinate prompt, the red-box prompt, and the RoI Align input all lose their grounding at once and the claimed controllability has no support in the paper.

Editorial extensions

If this is right

  • If the benchmark result holds, box-annotated video data is enough to steer open LVLMs toward intention-oriented captions, so the same recipe can be applied to other track-and-describe tasks without retraining a captioner from scratch.
  • The ablation numbers imply that coordinate prompts are the largest single prompt-side gain and that the red box is not redundant: each modality improves CIDEr on its own, but combining them in one model overfits at 214.45.
  • The adapter placement matters: inserting it in the last five ViT layers gives CIDEr 223.01, while eight or nine layers drop below 207, so the method's gain depends on choosing the right depth rather than adding more parameters.
  • The voting result implies that caption quality is improved by exploiting heterogeneity between models rather than trying to make one model absorb every prompt modality; the paper's simple short-video/long-video split already reaches 221.0 CIDEr before voting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the box adapter's cross-attention design is not caption-specific; the same global-local mechanism could be grafted onto other box-conditioned video tasks such as referring video segmentation or object-centric video editing, where a region prior must propagate through time.
  • Editorial inference: because the method needs ground-truth boxes at test time, its practical deployment depends on a tracker or detector supplying those boxes; testing with detector boxes rather than ground truth is the natural next experiment and is not covered in the paper.
  • Editorial inference: the overfitting observed when textual and visual prompts are merged suggests a testable variant: a single model might match the ensemble if the two prompt modalities are presented with random dropout during training, reducing prompt redundancy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes IntentVCNet, a framework for intention-oriented controllable video captioning on the IntentVC benchmark. The method combines textual coordinate prompts (per-frame normalized box coordinates in the instruction), visual red-box prompts drawn on video frames, and a parameter-efficient box adapter inserted into the visual encoder's last layers. The box adapter extracts region features via RoI alignment and injects them into global visual features through cross-attention. The vision encoder is frozen, the LLM is tuned with LoRA, and multiple heterogeneous models (InternVL3 and InternVideo2.5) are combined through a similarity-based voting mechanism. The paper reports CIDEr 225.19 on the IntentVC public test set and states that the method was runner-up in the IntentVC Challenge.

Significance. If the reported results are reproducible and the comparison is fair, the paper would make a useful empirical contribution: it demonstrates a parameter-efficient way to inject per-frame object-box information into open LVLMs for controllable video captioning, and it releases code, which supports reproducibility. The ablation study in Table 2 does show consistent gains from textual prompts and from the box adapter in several configurations. However, the central claim that the combination of textual prompts, visual prompts, and box adapter 'further enhances' performance is contradicted by the paper's own Table 2, and the headline 225.19 CIDEr result is not traceable to any fully described configuration. These issues are load-bearing for the paper's main assertion and need to be resolved before the contribution can be assessed.

major comments (4)
  1. [§4.5, Table 2] The full model TP+VP+BA achieves 217.17 CIDEr, which is 5.84 points below TP+BA (223.01) and 2.72 points below TP+VP (214.45) on the same table. The abstract's statement that 'the combination of the two strategies can further enhance' the LVLM's ability is therefore contradicted by the paper's own ablation. The explanation offered in the text ('redundancy... triggers overfitting') is plausible but is not backed by any experiment. Please either provide evidence that the final voting ensemble recovers this loss, or revise the central claim to reflect the actual behavior of the components.
  2. [§3.4 and §4.4] The headline result of 225.19 CIDEr in Table 1 cannot be traced to any configuration fully described in the paper. The best single-model configuration in Table 2 is 223.01 (TP+BA), and the simple fusion in Table 4 is 221.0. No experiment reports the voting ensemble's composition: which models were used, which prompt type was assigned to each model, what similarity metric was used, how many candidate sentences were generated, and how ties or disagreements were resolved. Please report the exact ensemble configuration and include an ablation of the voting mechanism so that the state-of-the-art claim is reproducible and falsifiable.
  3. [§4.4, Table 1] The four baselines in Table 1 are described only as 'fine-tuned on the IntentVC dataset.' If VAST, Qwen2.5-VL, InternVideo2.5, and InternVL3 did not receive the same per-frame coordinate and red-box prompt signals as IntentVCNet, the comparison is confounded: Table 2 shows that the prompt signals alone contribute roughly 14-15 CIDEr points over the 196.2 baseline. Please specify exactly which input signals each baseline received, and, if the baselines did not receive the box information, either retrain them with equivalent prompts or clearly label the comparison as 'ours with privileged box inputs versus baselines without them.'
  4. [§4.1 and §3.2] The method assumes per-frame ground-truth bounding boxes are available at test time, and the paper does not study robustness to noisy boxes, missing boxes, or settings where the user provides only a category or natural-language intent. Since the IntentVC benchmark supplies boxes, this is a scope limitation rather than an internal error, but the paper should state this limitation explicitly and, ideally, quantify how performance degrades when the box signal is corrupted or absent. The current wording of 'intention-oriented controllability' overstates the generality of the method.
minor comments (6)
  1. [Contributions, §1] The text says 'achieving outstanding performance with 225.19% CIDEr score'; CIDEr is a score, not a percentage, and should be written as '225.19 CIDEr.'
  2. [§4.1] The dataset description says 'each category contains 20 videos of different objects'; this should be clarified as 20 videos per category, and the total number of videos should be stated. The handling of the [0,0,0,0,0] absent-object convention also needs to be reconciled with the coordinate normalization described in §3.2.
  3. [§3.2] The sentence ending 'and the resulting user instructions.' is a grammatical fragment; please complete it and clarify how absent frames (zero boxes) are encoded in the coordinate list.
  4. [§3.3, Eq. (2)] The zero-conv operator Z is mentioned but not formally defined; please specify its input, output, and initialization in the equation or immediately after it.
  5. [Table 2] The first row of Table 2 has no checkmarks, which makes it easy to misread; please label it explicitly as the baseline or use a dash for absent components.
  6. [Fig. 2] The example coordinate list '267,261,816,228' appears without clarifying brackets or a caption explaining the format; please align the figure's notation with the four-coordinate convention [x1,y1,x2,y2] used in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the paper is an empirical system evaluated on the external IntentVC benchmark, and its main weaknesses are reproducibility/ablation-consistency issues rather than circularity.

full rationale

The paper does not present a first-principles derivation or a fitted parameter renamed as a prediction. It is a benchmark-driven system: the per-frame ground-truth boxes are task inputs supplied by the IntentVC dataset, and both the prompt combination (Sec. 3.2) and the box adapter (Eq. 1, RoI Align on bbox) condition on those boxes. That is task conditioning, not a prediction that reduces to its own inputs by construction. Table 2 ablated each component (TP, VP, BA), Table 3 ablated adapter depth, and Table 4 ablated simple fusion; these are empirical comparisons against fine-tuned baselines on the public test set, with no derivation chain whose conclusion is equivalent to an input. The only self-citation found is reference [50], a related-work citation on emotional video captioning, and it is not load-bearing. I also weighed the internal inconsistency flagged by the reviewing rule: the abstract claims 'the combination of the two strategies can further enhance' while Table 2 shows TP+VP+BA (217.17 CIDEr) underperforming TP+BA (223.01), and the headline 225.19 is not traceable to any described voting configuration. Those are correctness and reproducibility risks, not circularity, because no equation or definition makes the claimed result equal to its inputs. The external IntentVC challenge ranking provides an independent, non-circular anchor for the system's empirical status.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The ledger reflects that this is an empirical systems paper. Most assumptions are domain assumptions about the benchmark and about LVLM behavior, and the main hand-tuned choices are hyperparameters and the adapter-depth selection. No invented physical entities are introduced.

free parameters (6)
  • number of box-adapter layers = last 5 ViT layers
    Chosen from a sweep on the public test set (Table 3); last 5 gave CIDEr 223.01 versus 217.74 for the embed layer and 206.94 for last 8 layers.
  • LoRA rank = 128
    Set in Section 4.2 without a reported sweep; determines the size of the tunable subspace.
  • initial learning rate = 2e-5
    AdamW with cosine schedule in Section 4.2; chosen by hand, not ablated.
  • sampled frames per video = 32-48 train, 48 inference
    Random temporal sampling in Section 4.2; no ablation shown.
  • length split threshold for fusion = 74 frames
    Videos shorter than 74 frames go to InternVL and longer ones to InternVideo (Table 4); threshold chosen manually to exploit model comfort zones.
  • beam search length = 5
    Used for all inference in Section 4.5; no sensitivity analysis.
assumptions (4)
  • domain assumption Visual encoders of LVLMs are sensitive to salient visual markers such as colored boxes.
    Invoked in Section 3.2 and related work [44, 48] to justify red-box visual prompting; if false, visual prompts add no signal.
  • domain assumption The per-frame bounding boxes supplied in the IntentVC dataset are accurate and are available at test time as the user-intent signal.
    Section 4.1 states each frame has a unique grounding box; the method consumes these boxes in prompts and RoI alignment (Eq. 1), so box quality and availability are load-bearing.
  • domain assumption CIDEr, METEOR, BLEU-4, and ROUGE-L on the public test set measure intention-oriented caption quality.
    The benchmark defines these metrics in Section 4.3; the paper does not provide human evaluation or an intent-specific metric.
  • domain assumption Fine-tuning on the IntentVC training split and choosing hyperparameters on the public test split does not invalidate the reported test numbers.
    The box-adapter depth and fusion threshold were selected using public test results (Tables 3 and 4), so the final 225.19 number is partly a selected maximum rather than a single pre-registered configuration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning." pith.science (2026). https://pith.science/paper/NP3EVX7Y

@misc{pith2026250718531,
  author       = {Pith},
  title        = {Pith review of: IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NP3EVX7Y}},
  note         = {Machine review of arXiv:2507.18531}
}
read the original abstract

Intent-oriented controlled video captioning aims to generate targeted descriptions for specific targets in a video based on customized user intent. Current Large Visual Language Models (LVLMs) have gained strong instruction following and visual comprehension capabilities. Although the LVLMs demonstrated proficiency in spatial and temporal understanding respectively, it was not able to perform fine-grained spatial control in time sequences in direct response to instructions. This substantial spatio-temporal gap complicates efforts to achieve fine-grained intention-oriented control in video. Towards this end, we propose a novel IntentVCNet that unifies the temporal and spatial understanding knowledge inherent in LVLMs to bridge the spatio-temporal gap from both prompting and model perspectives. Specifically, we first propose a prompt combination strategy designed to enable LLM to model the implicit relationship between prompts that characterize user intent and video sequences. We then propose a parameter efficient box adapter that augments the object semantic information in the global visual context so that the visual token has a priori information about the user intent. The final experiment proves that the combination of the two strategies can further enhance the LVLM's ability to model spatial details in video sequences, and facilitate the LVLMs to accurately generate controlled intent-oriented captions. Our proposed method achieved state-of-the-art results in several open source LVLMs and was the runner-up in the IntentVC challenge. Our code is available on https://github.com/thqiu0419/IntentVCNet.

Figures

Figures reproduced from arXiv: 2507.18531 by the authors.

Figure 1
Figure 1. Comparisons between the traditional video cap [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of our framework for intention-oriented video captioning. 1) We first design a prompt combination, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RoadTones: Tone Controllable Text Generation from Road Event Videos

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Presents RoadTones-51K dataset, RoadTones-VL-CoT model with tone-conditioned CoT, and RoadTones-Eval suite for controllable tone in road video captioning, supported by user study.

Reference graph

Works this paper leans on

65 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  2. [2]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  3. [3]

    Qianwen Cao, Heyan Huang, and Boran Wang. 2025. From Skeleton to Flesh: Aggregated Relational Transformer Towards Controllable Video Captioning with Two-Step Decoding. In Proceedings of the 2025 International Conference on Multimedia Retrieval. 61–70

    work page 2025

  4. [4]

    Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. 2023. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292 (2023)

    arXiv 2023

  5. [5]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  6. [6]

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. 2023. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems 36 (2023), 72842–72866

    work page 2023

  7. [7]

    Tseng-Hung Chen, Kuo-Hao Zeng, Wan-Ting Hsu, and Min Sun. 2017. Video captioning via sentence augmentation and spatio-temporal attention. InComputer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I 13 . Springer, 269–286

    work page 2017

  8. [8]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    arXiv 2024

Show all 65 references
  1. [9]

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 Apri...

    2023

  2. [10]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv prepri...

    2020 arXiv

  3. [11]

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. 2019. LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking. InProceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2019

  4. [12]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799

    2019

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  6. [14]

    Kastner, Yasutomo Kawanishi, Trung Thanh Nguyen, and Junan Chen

    Takahiro Komamizu, Marc A. Kastner, Yasutomo Kawanishi, Trung Thanh Nguyen, and Junan Chen. 2025. IntentVC 2025: The ACM Multimedia Grand Challenge on Intention-Oriented Controllable Video Captioning. In Proceedings of the 33rd ACM International Conference on Multimedia . 1–2

    2025

  7. [15]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  8. [16]

    Xuelong Li, Bin Zhao, Xiaoqiang Lu, et al. 2017. MAM-RNN: Multi-level attention model based RNN for video captioning.. In IJCAI, Vol. 2017. 2208–2214

    2017

  9. [17]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: ...

    2021

  10. [18]

    Yiming Li and Zhao Zhang. 2024. The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA. arXiv:2402.18385 [cs.CL]

    2024 arXiv

  11. [19]

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. 2024. Groundinggpt: Language enhanced multi- modal grounding model. arXiv preprint arXiv:2401.06071 (2024)

    2024 arXiv

  12. [20]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    2023 arXiv

  13. [21]

    Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 17949–17958

    2022

  14. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  15. [23]

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093 (2023)

    2023 arXiv

  16. [24]

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. 2024. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision . Springer, 417–435

    2024

  17. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  18. [26]

    Ramakanth Pasunuru and Mohit Bansal. 2017. Reinforced video captioning with entailment rewards. arXiv preprint arXiv:1708.02300 (2017)

    2017 arXiv

  19. [27]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023)

    2023 arXiv

  20. [28]

    Lin CY ROUGE. 2004. A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain

    2004

  21. [29]

    Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. 2023. Audio-visual llm for video understanding. arXiv preprint arXiv:2312.06720 (2023)

    2023 arXiv

  22. [30]

    Peipei Song, Dan Guo, Jun Cheng, and Meng Wang. 2023. Contextual Attention Network for Emotional Video Captioning. IEEE Transactions on Multimedia 25 (2023), 1858–1867

    2023

  23. [31]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. 2024. Emo- tional Video Captioning With Vision-Based Emotion Interpretation Network. IEEE Transactions on Image Processing 33 (2024), 1122–1135

    2024

  24. [32]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, Erkun Yang, and Meng Wang

  25. [33]

    Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid

  26. [34]

    In Proceedings of the 31st ACM International Conference on Multimedia

    Emotion-Prior Awareness Network for Emotional Video Captioning. In Proceedings of the 31st ACM International Conference on Multimedia . 589–600

  27. [35]

    Yunlong Tang, Jing Bi, Chao Huang, Susan Liang, Daiki Shimada, Hang Hua, Yunzhong Xiao, Yizhi Song, Pinxin Liu, Mingqian Feng, et al . 2025. Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting. arXiv preprint arXiv:2504.055...

    2025 arXiv

  28. [36]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35 (2022), 10078–10093

    2022

  29. [37]

    Jiahui Sun, Peipei Song, Jing Zhang, and Dan Guo. 2024. Syntax-Controllable Video Captioning with Tree-Structural Syntax Augmentation. In Proceedings of the 2024 2nd Asia Conference on Computer Vision, Image Processing and Pattern Recognition. 1–7

    2024

  30. [38]

    Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu. 2019. Controllable video captioning with pos sequence guidance based on gated fusion network. In Proceedings of the IEEE/CVF international conference on computer vision. 2641–2650

    2019

  31. [39]

    Han Wang, Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. 2024. Elysium: Exploring object-level perception in videos via mllm. In European Conference on Computer Vision. Springer, 166–185

    2024

  32. [40]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

    2015

  33. [41]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al . 2024. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision. Springer, 396–416

    2024

  34. [42]

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al . 2025. InternVideo2. 5: Empowering Video MLLMs with Long and Rich Context Modeling.arXiv preprint arXiv:2501.12386 (2025)

    2025 arXiv

  35. [43]

    Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang

  36. [44]

    Jinheng Xie, Songhe Deng, Bing Li, Haozhe Liu, Yawen Huang, Yefeng Zheng, Jurgen Schmidhuber, Bernard Ghanem, Linlin Shen, and Mike Zheng Shou. 2024. Tune-an-ellipse: Clip has potential to find what you want. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  37. [45]

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. 2024. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15841 (2024)

    2024 arXiv

  38. [46]

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. 2022. Clip-vip: Adapting pre-trained image-text model to video- language representation alignment. arXiv preprint arXiv:2209.06430 (2022). MM ’25, October 27-October 31, 2025, Dublin, Irel...

    2022 arXiv

  39. [47]

    Xinlong Xiao, Yuejie Zhang, Rui Feng, Tao Zhang, Shang Gao, and Weiguo Fan

  40. [48]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao

  41. [49]

    Linli Yao, Yuanmeng Zhang, Ziheng Wang, Xinglin Hou, Tiezheng Ge, Yuning Jiang, Xu Sun, and Qin Jin. 2024. Edit As You Wish: Video Caption Editing with Multi-grained User Control. In Proceedings of the 32nd ACM International Conference on Multimedia. 1924–1933

    2024

  42. [50]

    Cheng Ye, Weidong Chen, Jingyu Li, Lei Zhang, and Zhendong Mao. 2024. Dual- path collaborative generation network for emotional video captioning. In Pro- ceedings of the 32nd ACM International Conference on Multimedia . 496–505

    2024

  43. [51]

    Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. 2023. Hitea: Hierarchical temporal-aware video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 15405– 15416

    2023

  44. [52]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)

    2025 arXiv

  45. [53]

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. 2025. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprint arXiv:2501.13106 (2025)

    2025 arXiv

  46. [54]

    arXiv preprint arXiv:2310.11441 (2023)

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023)

    2023 arXiv

  47. [55]

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- trol to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision . 3836–3847

    2023

  48. [56]

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. 2025. Gpt4roi: Instruction tuning large language model on region-of-interest. In European conference on computer vision . Springer, 52–70

    2025

  49. [57]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  50. [58]

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 (2023)

    2023 arXiv

  51. [59]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025)

    2025 arXiv

  52. [60]

    Junchao Zhang and Yuxin Peng. 2019. Object-aware aggregation with bidi- rectional temporal graph for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 8327–8336

    2019

  53. [64]

    Fangyi Zhu, Jenq-Neng Hwang, Zhanyu Ma, Guang Chen, and Jun Guo. 2020. Ovc-net: Object-oriented video captioning with temporal graph and detail en- hancement. arXiv preprint arXiv:2003.03715 (2020)

    2020 arXiv

  54. [2018]

    In Proceedings of the IEEE conference on computer vision and pattern recognition

    Video captioning via hierarchical reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4213–4222

  55. [2019]

    In Proceedings of the IEEE/CVF international conference on computer vision

    Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision . 7464– 7473

  56. [2020]

    In 2020 IEEE International Conference on Multimedia and Expo (ICME)

    Video captioning with temporal and region graph convolution network. In 2020 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 1–6

    2020

  57. [2023]

    arXiv preprint arXiv:2306.15195 (2023)

    Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.