Pith. sign in

REVIEW 2 major objections 6 minor 5 cited by

Number it: Temporal Grounding Videos like Flipping Manga

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Printing each frame's index onto the video lets Vid-LLMs read off event timestamps instead of hallucinating them, with no retraining.

desk verdict A simple, plausible trick for VTG that deserves a careful referee; the missing instruction-only control is the one thing I'd ask for before trusting the training-free numbers. read the letter →

arxiv 2411.10332 v3 pith:3TN6ZRNZ submitted 2024-11-15 cs.CV

Yongliang Wu , Xinting Hu , Yuyang Sun , Yizhou Zhou , Wenbo Zhu , Fengyun Rao , Bernt Schiele , Xu Yang This is my paper · ORCID
classification cs.CV
keywords videotemporalgroundingframe-numberoverlayvisualpromptinglargelanguagemodelsmomentretrievalhighlightdetectionOCRtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-language models (Vid-LLMs) can say what happens in a video but not when: asked for an event's timing, they hallucinate frame ranges that are impossible for the actual footage. This paper claims the bottleneck is not visual recognition but verbalization — attention maps show the model already fixes its gaze on the right frames — and that a simple visual prompt closes the gap. NumPro prints each frame's index onto the frame itself, so the model's built-in optical character recognition can read the timeline like numbered manga panels and answer with exact frame numbers. The paper shows this overlay lifts temporal-grounding accuracy across several top Vid-LLMs with no training, and that fine-tuning on number-augmented data establishes a new state of the art on moment retrieval and highlight detection. A sympathetic reader would care because the fix is a two-line preprocessing step rather than a new architecture.

What carries the argument

The load-bearing object is the numbered frame: each input image carries its index printed in red at the bottom right, making the video's timeline part of the visual signal the model already processes. The mechanism that makes it work is OCR — Vid-LLMs are trained on image-text data and can read printed digits, so the frame index is recovered by perception rather than synthesized from a learned temporal mapping. Three supporting pieces carry the argument: an attention-map analysis (Qwen2-VL attends to the correct frames yet emits impossible timestamps) that identifies verbalization, not localization, as the failure; a CLIP-based design search over font size, color, and position that balances Number Accuracy against Caption Accuracy on MSCOCO images; and a fine-tuning stage (NumPro-FT) that freezes the visual encoder, applies LoRA to the LLM, and trains on a number-augmented instruction dataset so the model learns to emit answers in the form 'From x to y' in frame units.

What would settle it

Run the Charades-STA moment-retrieval queries through Qwen2-VL-7B with the NumPro prompt sentence but with the frame numbers omitted or rendered invisible (for example, red digits on a red patch the model classes as background). If mIoU stays near the reported 38.5 instead of falling back toward the 7.9 baseline, the overlay itself is not the operative cause and the OCR-reading explanation would need to be revised.

Watch

Extended reading notes

Core claim

The paper's central claim is that numerical identifiers overlaid on video frames convert Video Temporal Grounding from a generation problem into a reading problem. Qwen2-VL's attention maps already point at the query-relevant frames; what fails is translating that attention into textual start and end times, so the model emits boundaries like 'from 000 to 580' for a ten-frame video. NumPro adds one red frame number (font size 40, bottom-right, selected by a CLIP-based trade-off between number readability and caption preservation) to every frame and instructs the model that the red numbers are frame numbers. The model then reports boundaries directly from the visual input — answers such as 'from 3 to 12' — and in the training-free setting Qwen2-VL-7B's mIoU rises from 7.9 to 38.5 on Charades-STA. Fine-tuning LongVA-7B-DPO with LoRA on a 220k-sample number-augmented instruction dataset yields new state-of-the-art results, surpassing prior VTG-tuned methods by up to 6.9% in mIoU for moment retrieval and 8.5% in mAP for highlight detection. The paper concludes that the overlay, not model surgery, is the operative intervention: it works across open- and closed-source models, transfers to highlight detection and general video QA with minimal side effects, and needs no additional computational cost at inference.

Load-bearing premise

The load-bearing premise is that the improvement comes from the visible numbers, because every numbered-frame condition also adds the sentence 'The red numbers on each frame represent the frame number' that the unnumbered baseline never receives, and no control runs that sentence with unnumbered frames — so part or all of the measured gain could come from the instruction alone.

Editorial extensions

If this is right

  • Any current Vid-LLM with basic OCR can gain temporal grounding as a training-free upgrade: moment retrieval and highlight detection improve without new parameters or extra inference cost.
  • The gains transfer across model families — GPT-4o, Qwen2-VL-7B/72B, LLaVA-Video, LLaVA-OneVision — and across benchmarks (Charades-STA, ActivityNet, QVHighlights).
  • Fine-tuning on a number-augmented dataset (NumPro-FT) sets a new state of the art, surpassing prior VTG-tuned methods by up to 6.9% in mIoU for moment retrieval and 8.5% in mAP for highlight detection.
  • Numbering only a fraction of frames still helps: uniformly overlaying numbers on 80% of frames already beats previous state-of-the-art on moment retrieval.
  • General video QA is largely preserved (VideoInstruct, MVBench, VideoMME), with temporal-understanding sub-scores improving slightly, so the overlay can be left on in mixed conversational video systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A control the paper does not report — giving the same instruction sentence ('The red numbers on each frame represent the frame number') on frames with no numbers, or with numbers printed invisibly — would separate the overlay's contribution from the instruction's. If the instruction alone recovers much of the gain, the 'OCR reads the timeline' story is only part of the explanation.
  • If the OCR mechanism is really what carries the improvement, gains should track the model's text-reading ability: tiny fonts, low contrast, or non-digit glyphs should erode the effect, and models with weak OCR should benefit less. The paper's ablations touch readability but do not test this gradient directly.
  • The recipe is general: any quantity a video model knows implicitly but cannot verbalize — shot indices, saliency scores, object tracks — could be overlaid rather than injected into the token vocabulary, turning other 'cannot say when or where' failures into reading tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces Number-Prompt (NumPro), a visual-prompting method that overlays unique frame numbers onto video frames. In the training-free variant, the frame numbers are read by Vid-LLMs via OCR after a textual instruction is prepended; in the fine-tuned variant, a LoRA-tuned LLM is trained on a NumPro-augmented instruction dataset (NumPro-FT). The authors report large improvements on Charades-STA, ActivityNet, and QVHighlights across GPT-4o, Qwen2-VL-7B/72B, LLaVA-Video, LLaVA-OneVision, and LongVA, and claim new state-of-the-art results for moment retrieval and highlight detection. They also contribute a CLIP-based design analysis for font size, color, and position, an attention analysis of why Vid-LLMs struggle with temporal grounding, and ablations on sampling ratio and sampling strategy.

Significance. If the causal attribution holds, this is an elegant and potentially widely applicable idea: it converts timestamp prediction into a visual reading task, requiring no architectural changes, no new tokens, and no changes to the model vocabulary. The paper's strengths include the breadth of models and benchmarks tested, the transferable CLIP-based design methodology, the attention analysis in Section 3.1 and Appendix 8, and the planned code release. However, the central causal claim that the number overlay itself, rather than the accompanying textual instruction, produces the gains is not yet isolated, and the fine-tuning SOTA claim needs clarification about possible train/eval overlap. These issues are fixable but currently leave the headline conclusions under-supported.

major comments (2)
  1. [Section 3.2, Tables 1-2] The training-free comparison conflates two interventions. Section 3.2 states that with NumPro the authors prepend the instruction 'The red numbers on each frame represent the frame number.' The baseline receives neither the overlay nor this instruction, so the observed gains (Qwen2-VL-7B mIoU on Charades-STA from 7.9 to 38.5 in Table 1; LLaVA-Video-7B from 9.8 to 34.6 in Table 2) could be partly or wholly attributable to the instruction alone. An instruction-only control, that is, the same prompt applied to unnumbered frames, must be reported. Table 3's design ablations hold the instruction fixed and show that overlay properties matter, but they do not quantify the instruction's contribution. Without this control, the abstract's claim that the number overlay 'significantly boosts VTG performance' is not established.
  2. [Section 4.1, Section 4.2, Tables 1-2] The NumPro-FT training set is built from DiDeMo and ActivityNet Caption data, while the headline moment-retrieval evaluation is on ActivityNet. The paper never states whether the ActivityNet evaluation videos were excluded from the 220k training samples. If the same video instances, or their temporal annotations, appear in both training and evaluation, the SOTA gains in Table 1 would be inflated. Please clarify the exact train/eval split, and if necessary retrain without the evaluation split and report the comparison. This is essential for the fine-tuning contribution, whose purpose is to show that number-augmented training data adds value beyond conventional fine-tuning.
minor comments (6)
  1. [Tables 1-2] The '+FT' (fine-tuning without NumPro) baseline appears in Table 2 but is omitted from Table 1. Since Table 2 shows that +FT already reaches 40.2 mIoU on Charades-STA, the contribution of the number overlay in the fine-tuned setting should be made more visible in the main comparison table.
  2. [Section 3.2, Figure 3] Please clarify whether the instruction 'The red numbers on each frame represent the frame number' is also used in the fine-tuning stage and in the '+FT' baseline. If the instruction differs between +FT and +NumPro-FT, that difference is an additional confound in the fine-tuning comparison.
  3. [Tables 1-3 and Appendix 10] The tables report single runs without error bars or seed information. Given the very large reported gains for some models, such as Qwen2-VL-72B moving from near zero to 17.4 mIoU, reporting variance across at least three seeds would substantially strengthen the results.
  4. [Appendix 12, Table 9] The comparison between timestamp overlays and frame-number overlays is difficult to parse because the table rows are not labeled. Please specify which rows correspond to which overlay format and ensure the units (seconds, minutes, or frame indices) are stated in the table header.
  5. [References and notation] There are several presentation issues: references [8] and [9] appear to be the same paper; 'Tabel 9' should be 'Table 9'; and the model names are inconsistently spaced (e.g., 'LLaV A-Video'). These should be corrected.
  6. [Abstract, Section 3.2] The phrase 'without additional computational cost' should be qualified: the method requires the user to pre-render frame-number overlays and to treat the overlaid video as the model input. That is an input-modality assumption rather than a strictly cost-free zero-shot capability, and the paper should state this clearly.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: NumPro's frame-number overlay is the proposed visual coordinate system rather than a hidden reuse of the answer, and the core claims are tested on external VTG benchmarks. The missing instruction-only control is a real attribution confound, but it is not a circular step.

full rationale

No significant circularity is established. The paper's chain is empirical rather than derivational: Sec. 3.1 and Appendix Table 7 show Qwen2-VL-7B's attention already selects query-relevant frames, and NumPro then overlays frame numbers so those frames can be verbalized as an interval. The overlaid numbers are a coordinate system, not the ground-truth event interval; the model still must choose which numbered frames bound the event, so the target output is not identical to the input by construction. The only self-citations (Refs. 72 and 74) are background/related-work mentions with no load-bearing role, and no uniqueness theorem or ansatz is imported from the authors' prior work. The paper is self-contained against external VTG benchmarks, which further supports a low score. The genuine caveat is a missing instruction-only control: Sec. 3.2 prepends 'The red numbers on each frame represent the frame number' only in the NumPro condition, so part of the training-free gain could in principle come from the instruction rather than the overlay; this is a confound/attribution risk, not a circular step. Because the alleged identity (output = overlaid label) is the method itself rather than a hidden reuse of the answer, the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on a set of unverified assumptions: OCR reliability, instruction-only neutrality, disjoint train/test data, and transfer from CLIP design tests. The free parameters are design choices (font, color, position, frame rate) selected by hand/CLIP rather than derived.

free parameters (4)
  • Font size of frame number overlay = 40
    Selected via CLIP-based Number/Caption accuracy on COCO; validated on Charades-STA. Size 20 hurts readability, 60/80 hurt caption accuracy (Sec 3.3, Fig 5, Table 3).
  • Number color = red
    Red gave best number accuracy in CLIP scan; black worst (Sec 3.3, Fig 5).
  • Number position = bottom-right
    Bottom-right balances number and caption accuracy; center hurts caption accuracy (Sec 3.3, Fig 5).
  • Frame sampling rate = 1 FPS (training-free), 0.5 FPS (fine-tuned)
    Fixed choice; authors list frame-rate optimization as an underexplored limitation (Sec 6, Sec 14).
assumptions (4)
  • domain assumption Vid-LLMs can reliably OCR the overlaid frame numbers at 336x336 resolution and associate them with visual content.
    Stated in Sec 3.2; the method relies on built-in OCR. CLIP-based design selection approximates but does not directly test OCR on Vid-LLMs.
  • ad hoc to paper The added textual instruction ('The red numbers...') is not responsible for the improvement.
    No control condition without numbers measures the instruction alone (Sec 3.2).
  • domain assumption The NumPro-FT training mixture (DiDeMo, ActivityNet Caption, VTimeLLM stages) does not include the evaluation splits of ActivityNet/Charades-STA.
    Sec 4.1 does not clarify split exclusion; if false, reported SOTA is inflated.
  • domain assumption CLIP-based design choices on COCO still images transfer to video frames and Vid-LLMs.
    The paper validates this on Charades-STA in Sec 4.3, so it is partially supported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Number it: Temporal Grounding Videos like Flipping Manga." pith.science (2026). https://pith.science/paper/3TN6ZRNZ

@misc{pith2026241110332,
  author       = {Pith},
  title        = {Pith review of: Number it: Temporal Grounding Videos like Flipping Manga},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TN6ZRNZ}},
  note         = {Machine review of arXiv:2411.10332}
}
read the original abstract

Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro.

Figures

Figures reproduced from arXiv: 2411.10332 by the authors.

Figure 1
Figure 1. Effectiveness of Adding Frame Numbers for Temporal Grounding: (a) Without numbered images or frames, both humans and Vid-LLMs struggle to locate specific timestamps accurately. (b) Once numbered, grounding temporal cues becomes as intuitive as flipping manga, where timestamps are accessible at a glance. Abstract Video Large Language Models (Vid-LLMs) have made re￾markable advancements in comprehending video content … view at source ↗
Figure 2
Figure 2. Attention Analysis between Video Frames and Event Query. Although the model accurately attends to regions of inter￾est related to the query, it struggles to generate precise temporal boundaries in its response. ity across various tasks and datasets. Temporal ground￾ing, therefore, becomes an accessible, “free-lunch” en￾hancement for Vid-LLMs already proficient in understand￾ing video content. Additionally, fine-tuni… view at source ↗
Figure 3
Figure 3. Framework of Our Approach in Two Settings: (1) Training-free VTG with NumPro, where frame numbers are directly added to video frames, enabling Vid-LLMs to locate events temporally without additional training, and (2) Fine-tuned VTG with NumPro-FT, which further improves VTG performance by fine-tuning Vid-LLMs on a dataset NumPro-enhanced with no architectural modifications. cific temporal embeddings [20, 58]. Additi… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Our NumPro Design Algorithm. We overlay dif￾ferent numbers onto COCO images and obtain visual and textual representations using CLIP encoders. For each configuration, we calculate Number/Caption Similarity and derive Number/Caption Accuracy, to identify the optimal Num…
Figure 5
Figure 5. Figure 5: The Impact of Different Number-Prompt Designs. We categorize the design into three dimensions: font size, position, and color. BL stands for Bottom Left, BR for Bottom Right, TL for Top Left, TR for Top Right, and C for Center. ing the font size improves number accurac…
Figure 6
Figure 6. Figure 6: Qualitative Comparison with State-of-the-Art. Our LongVA-7B-DPO model, fine-tuned with NumPro-FT, outperforms TimeChat [58] and VTimeLLM [25] on ActivityNet by accurately identifying event boundaries in challenging scenes. 0.0 0.2 0.4 0.6 0.8 1.0 Ratio 10 20 30 40 mIoU…
Figure 7
Figure 7. Figure 7: Performance Comparison of Sampling Strategies for NumPro. We compare the effects of NumPro with different sam￾pling ratios and sampling methods (random vs. uniform), as tested on the Charades-STA [18] using the Qwen2-VL-7B [66] model. to harness a powerful Vid-LLM for …
Figure 8
Figure 8. Figure 8: Video temporal grounding results where the models exhibit serious hallucination and output incorrect results. In all cases, frames [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Distribution of the Top 10 Most Common Time Intervals within the Charades-STA Dataset for Different Models. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Additional attention analysis results on general Vid-LLMs [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: An example of multi-turn dialogue using our NumPro method in the Qwen2-VL-7B model. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Additional visualization cases of Video Temporal Grounding task. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Additional visualization cases of Highlight Detection task on QVHighlights dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TimePLE: Rethinking Temporal Representation for Video Temporal Grounding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    TimePLE predicts a whole video interval as a joint distribution over a position-duration square, rather than predicting start and end separately, and reports higher mIoU across four VTG benchmarks.

  2. ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ViaRL uses rule-based reinforcement learning to train a frame selector for video QA, improving Qwen2.5-VL on VideoMME, LVBench, and MLVU by several points.

  3. VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism

    cs.CV 2025-06 conditional novelty 5.0 of 10

    VReST combines Monte Carlo tree search with a self-reward signal inside a vision-language model to get higher accuracy than CoT, ToT, or voting baselines on MathVista, MathVision, and CharXiv, while spending several t...

  4. RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RSVP couples region-grid visual prompting and multimodal chain-of-thought reasoning with a BEiT-3/SAM segmentation module, achieving state-of-the-art zero-shot results on ReasonSeg and SegInW.

  5. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

Reference graph

Works this paper leans on

101 extracted references · 36 canonical work pages · cited by 5 Pith papers

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 2

    arXiv 2023

  2. [2]

    Rasa: Relation and sensitivity aware representation learning for text-based person search

    Yang Bai, Min Cao, Daming Gao, Ziqiang Cao, Chen Chen, Zhenfeng Fan, Liqiang Nie, and Min Zhang. Rasa: Relation and sensitivity aware representation learning for text-based person search. arXiv preprint arXiv:2305.13653, 2023. 3

    arXiv 2023

  3. [3]

    The surprising effectiveness of multimodal large language models for video moment retrieval

    Meinardus Boris, Batra Anil, Rohrbach Anna, and Rohrbach Marcus. The surprising effectiveness of multimodal large language models for video moment retrieval. arXiv preprint arXiv:2406.18113, 2024. 2

    arXiv 2024

  4. [4]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 2, 5

    2015

  5. [5]

    Vip- llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip- llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12914– 12923, 2024. 3, 4, 5

    2024

  6. [6]

    Progressive bilateral-context driven model for post-processing person re-identification

    Min Cao, Chen Chen, Hao Dou, Xiyuan Hu, Silong Peng, and Arjan Kuijper. Progressive bilateral-context driven model for post-processing person re-identification. IEEE Transactions on Multimedia, 23:1239–1251, 2020. 3

    2020

  7. [7]

    Image-text retrieval: A survey on recent research and development

    Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.14713, 2022. 2

    arXiv 2022

  8. [8]

    An empirical study of clip for text-based person search

    Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, and Min Zhang. An empirical study of clip for text-based person search. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 465–473, 2024. 4

    2024

Show all 101 references
  1. [9]

    An empirical study of clip for text-based person search

    Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, and Min Zhang. An empirical study of clip for text-based person search. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 465–473, 2024. 2

    2024

  2. [10]

    End- to-end multi-modal video temporal grounding

    Yi-Wen Chen, Yi-Hsuan Tsai, and Ming-Hsuan Yang. End- to-end multi-modal video temporal grounding. Advances in Neural Information Processing Systems , 34:28442–28453,

  3. [11]

    Instructdet: Diversifying referring ob- ject detection with generalized instructions

    Ronghao Dang, Jiangyan Feng, Haodong Zhang, Chongjian Ge, Lin Song, Lijun Gong, Chengju Liu, Qijun Chen, Feng Zhu, Rui Zhao, et al. Instructdet: Diversifying referring ob- ject detection with generalized instructions. arXiv preprint arXiv:2310.05136, 2023. 3

    arXiv

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 4

    2010 arXiv

  5. [13]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691, 2024. 4

    2024 arXiv

  6. [14]

    Cityllava: Efficient fine-tuning for vlms in city scenario

    Zhizhao Duan, Hao Cheng, Duo Xu, Xi Wu, Xiangxie Zhang, Xi Ye, and Zhen Xie. Cityllava: Efficient fine-tuning for vlms in city scenario. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7180–7189, 2024. 3

    2024

  7. [15]

    System- status-aware adaptive network for online streaming video un- derstanding

    Lin Geng Foo, Jia Gong, Zhipeng Fan, and Jun Liu. System- status-aware adaptive network for online streaming video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10514– 10523, 2023. 2

    2023

  8. [16]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 4, 8, 2

    2024 arXiv

  9. [17]

    Fast video moment re- trieval

    Junyu Gao and Changsheng Xu. Fast video moment re- trieval. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 1523–1532, 2021. 2

    2021

  10. [18]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on com- puter vision, pages 5267–5275, 2017. 2, 5, 7, 8

    2017

  11. [19]

    Scaling new frontiers: In- sights into large recommendation models

    Wei Guo, Hao Wang, Luankang Zhang, Jin Yao Chin, Zhongzhou Liu, Kai Cheng, Qiushi Pan, Yi Quan Lee, Wanqi Xue, Tingjia Shen, et al. Scaling new frontiers: In- sights into large recommendation models. arXiv preprint arXiv:2412.00714, 2024. 2

    2024 arXiv

  12. [20]

    Vtg-llm: Integrating timestamp knowl- edge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Xi Chen, and Bo Zhao. Vtg-llm: Integrating timestamp knowl- edge into video llms for enhanced video temporal grounding. arXiv preprint arXiv:2405.13382, 2024. 2, 3, 5, 6

    2024 arXiv

  13. [21]

    Trace: Temporal grounding video llm via causal event modeling.arXiv preprint arXiv:2410.05643,

    Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Qing- bin Liu, and Xi Chen. Trace: Temporal grounding video llm via causal event modeling.arXiv preprint arXiv:2410.05643,

    arXiv

  14. [22]

    Homeomorphism prior for false posi- tive and negative problem in medical image dense contrastive representation learning

    Yuting He, Boyu Wang, Rongjun Ge, Yang Chen, Guanyu Yang, and Shuo Li. Homeomorphism prior for false posi- tive and negative problem in medical image dense contrastive representation learning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 2025. 2

    2025

  15. [23]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 2

    2024 arXiv

  16. [24]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2021. 4

    2021

  17. [25]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14271–14280, 2024. 2, 5, 6, 8, 1

    2024

  18. [26]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. arXiv preprint arXiv:2403.19046, 2024. 2, 3, 6

    2024 arXiv

  19. [27]

    Dg- pic: Domain generalized point-in-context learning for point cloud understanding

    Jincen Jiang, Qianyu Zhou, Yuhang Li, Xuequan Lu, Meili Wang, Lizhuang Ma, Jian Chang, and Jian Jun Zhang. Dg- pic: Domain generalized point-in-context learning for point cloud understanding. In European Conference on Computer Vision, pages 455–474. Springer, 2024. 4

    2024

  20. [28]

    Do you remember? dense video captioning with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13904, 2024. 2

    2024

  21. [29]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 5

    2014 arXiv

  22. [30]

    Multi-scale spatial-temporal attention networks for functional connectome classification

    Youyong Kong, Xiaotong Zhang, Wenhan Wang, Yue Zhou, Yueying Li, and Yonggui Yuan. Multi-scale spatial-temporal attention networks for functional connectome classification. IEEE Transactions on Medical Imaging, 2024. 4

    2024

  23. [31]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on com- puter vision, pages 706–715, 2017. 2

    2017

  24. [32]

    Collavo: Crayon large language and vision model

    Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. Collavo: Crayon large language and vision model. arXiv preprint arXiv:2402.11248, 2024. 3

    2024 arXiv

  25. [33]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34: 11846–11858, 2021. 2, 5, 3

    2021

  26. [34]

    Frame order matters: A temporal sequence-aware model for few-shot action recognition

    Bozheng Li, Mushui Liu, Gaoang Wang, and Yunlong Yu. Frame order matters: A temporal sequence-aware model for few-shot action recognition. arXiv preprint arXiv:2408.12475, 2024. 2

    2024 arXiv

  27. [35]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 6, 7, 1

    2024 arXiv

  28. [36]

    Llava-st: A multimodal large language model for fine-grained spatial- temporal understanding

    Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained spatial- temporal understanding. arXiv preprint arXiv:2501.08282 ,

    arXiv

  29. [37]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  30. [38]

    Learning semantic- aligned feature representation for text-based person search

    Shiping Li, Min Cao, and Min Zhang. Learning semantic- aligned feature representation for text-based person search. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2724–2728. IEEE, 2022. 3

    2022

  31. [39]

    Tea: Temporal excitation and aggregation for action recognition

    Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 909–918, 2020. 2

    2020

  32. [40]

    Groundinggpt: Language enhanced multi-modal grounding model

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. Groundinggpt: Language enhanced multi-modal grounding model. arXiv preprint arXiv:2401.06071, 2024. 2, 6

    2024 arXiv

  33. [41]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2794–...

    2023

  34. [42]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  35. [43]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  36. [44]

    Omniclip: Adapt- ing clip for video recognition with spatial-temporal omni- scale feature learning

    Mushui Liu, Bozheng Li, and Yunlong Yu. Omniclip: Adapt- ing clip for video recognition with spatial-temporal omni- scale feature learning. arXiv preprint arXiv:2408.06158 ,

    arXiv

  37. [45]

    General- ized video anomaly event detection: Systematic taxonomy and comparison of deep models

    Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng Sun, and Liang Song. General- ized video anomaly event detection: Systematic taxonomy and comparison of deep models. ACM Computing Surveys, 56(7):1–38, 2024. 2

    2024

  38. [46]

    Zero-shot model diagnosis

    Jinqi Luo, Zhaoning Wang, Chen Henry Wu, Dong Huang, and Fernando De la Torre. Zero-shot model diagnosis. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 11631–11640, 2023. 4

    2023

  39. [47]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiao- juan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417–435. Springer, 2025. 3

    2025

  40. [48]

    Visual percep- tion by large language model’s weights

    Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual percep- tion by large language model’s weights. arXiv preprint arXiv:2405.20339, 2024. 3

    2024 arXiv

  41. [49]

    Invari- ant test-time adaptation for vision-language model general- ization

    Huan Ma, Yan Zhu, Changqing Zhang, Peilin Zhao, Baoyuan Wu, Long-Kai Huang, Qinghua Hu, and Bingzhe Wu. Invari- ant test-time adaptation for vision-language model general- ization. arXiv preprint arXiv:2403.00376, 2024. 3

    2024 arXiv

  42. [50]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 7, 2

    2023 arXiv

  43. [51]

    Weakly supervised video moment retrieval from text queries

    Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy- Chowdhury. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11592– 11601, 2019. 5

    2019

  44. [52]

    Interventional video ground- ing with dual contrastive learning

    Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu. Interventional video ground- ing with dual contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2765–2775, 2021. 2

    2021

  45. [53]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https : / / openai . com / index/hello-gpt-4o/ , 2024. Accessed: 2024-10-21. 6

    2024

  46. [54]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025. 2

    2025 arXiv

  47. [55]

    Momen- tor: Advancing video large language model with fine-grained temporal reasoning

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat- Seng Chua, Yueting Zhuang, and Siliang Tang. Momen- tor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402.11435, 2024. 2, 5, 6

    2024 arXiv

  48. [56]

    Controllable augmentations for video representation learning

    Rui Qian, Weiyao Lin, John See, and Dian Li. Controllable augmentations for video representation learning. Visual In- telligence, 2(1):1, 2024. 2

    2024

  49. [57]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  50. [58]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14313–14323, 2024. 2, 3, 5, 6, 8, 1

    2024

  51. [59]

    Towards more unified in-context visual un- derstanding

    Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Tao Gong, Bin Liu, Shengwei Xu, and Nenghai Yu. Towards more unified in-context visual un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13362...

    2024

  52. [60]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11987–11997, 2023. 3, 5, 7

    2023

  53. [61]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  54. [62]

    Add-it: Training-free object inser- tion in images with pretrained diffusion models, 2024

    Yoad Tewel, Rinon Gal, Dvir Samuel, Yuval Atzmon, Lior Wolf, and Gal Chechik. Add-it: Training-free object inser- tion in images with pretrained diffusion models, 2024. 3

    2024

  55. [63]

    Towards open- world grasping with large vision-language models

    Georgios Tziafas and Hamidreza Kasaei. Towards open- world grasping with large vision-language models. arXiv preprint arXiv:2406.18722, 2024. 3

    2024 arXiv

  56. [64]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024. 2

    arXiv

  57. [65]

    Temporal segment networks for action recognition in videos.IEEE transactions on pattern analysis and machine intelligence , 41(11):2740– 2755, 2018

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos.IEEE transactions on pattern analysis and machine intelligence , 41(11):2740– 2755, 2018. 2

    2018

  58. [66]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 4, 6, 7, 8, 1

    2024 arXiv

  59. [67]

    Visual- semantic network: a visual and semantic enhanced model for gesture recognition

    Yizhe Wang, Congqi Cao, and Yanning Zhang. Visual- semantic network: a visual and semantic enhanced model for gesture recognition. Visual Intelligence, 1(1):25, 2023. 2

    2023

  60. [68]

    Hawkeye: Training video- text llms for grounding text in videos

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video- text llms for grounding text in videos. arXiv preprint arXiv:2403.10228, 2024. 2, 3, 5, 6

    2024 arXiv

  61. [69]

    Omniedit: Building image editing generalist models through specialist supervision, 2024

    Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, and Wenhu Chen. Omniedit: Building image editing generalist models through specialist supervision, 2024. 3

    2024

  62. [70]

    Visual prompting in multi- modal large language models: A survey

    Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multi- modal large language models: A survey. arXiv preprint arXiv:2409.15310, 2024. 3

    2024 arXiv

  63. [71]

    Dora: 3d visual grounding with order-aware referring

    Tung-Yu Wu, Sheng-Yu Huang, and Yu-Chiang Frank Wang. Dora: 3d visual grounding with order-aware referring. arXiv preprint arXiv:2403.16539, 2024. 3

    2024 arXiv

  64. [72]

    A glance at in-context learning

    Yongliang Wu and Xu Yang. A glance at in-context learning. Frontiers of Computer Science, 18(5):185347, 2024. 3

    2024

  65. [73]

    Det- toolchain: A new prompting paradigm to unleash detection ability of mllm

    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Jian Wu, and Philip Torr. Det- toolchain: A new prompting paradigm to unleash detection ability of mllm. arXiv preprint arXiv:2403.12488, 2024. 3

    2024 arXiv

  66. [74]

    Video repurposing from user generated con- tent: A large-scale dataset and benchmark

    Yongliang Wu, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, et al. Video repurposing from user generated con- tent: A large-scale dataset and benchmark. arXiv preprint arXiv:2412.08879, 2024. 4

    2024 arXiv

  67. [75]

    Sutd-trafficqa: A question answering benchmark and an efficient network for video rea- soning over traffic events

    Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video rea- soning over traffic events. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9878–9888, 2021. 2

    2021

  68. [76]

    Meta spatio-temporal debiasing for video scene graph gen- eration

    Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu. Meta spatio-temporal debiasing for video scene graph gen- eration. In European Conference on Computer Vision, pages 374–390. Springer, 2022. 2

    2022

  69. [77]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. InProceed- ings of the IEEE/CVF Conference on Computer Vi...

    2023

  70. [78]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 3

    2023 arXiv

  71. [79]

    Exploring diverse in-context configurations for image captioning

    Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3

    2024

  72. [80]

    Cpt: Colorful prompt tuning for pre-trained vision-language models

    Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat- Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. AI Open, 5:30–38,

  73. [81]

    Visual place recognition via local affine preserving matching

    Xinyu Ye and Jiayi Ma. Visual place recognition via local affine preserving matching. In2021 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 12954– 12960. IEEE, 2021. 2

    2021

  74. [82]

    Neighborhood manifold preserving matching for visual place recognition

    Xinyu Ye and Jiayi Ma. Neighborhood manifold preserving matching for visual place recognition. IEEE Transactions on Industrial Informatics, 19(7):8127–8136, 2022

    2022

  75. [83]

    Vqne: Variational quan- tum network embedding with application to network align- ment

    Xinyu Ye, Ge Yan, and Junchi Yan. Vqne: Variational quan- tum network embedding with application to network align- ment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3105– 3115, 2023. 2

    2023

  76. [84]

    Bridge the modality and capability gaps in vision-language model selection

    Chao Yi, Yuhang He, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capability gaps in vision-language model selection. Advances in Neural Information Processing Systems, 37:34429–34452, 2024. 2

    2024

  77. [85]

    Dataset regeneration for sequential recommendation

    Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. Dataset regeneration for sequential recommendation. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3954–3965, 2024. 2

    2024

  78. [86]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Bar- las Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 23056–23065, 2023. 2

    2023

  79. [87]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 5, 6, 7

    2024 arXiv

  80. [88]

    Pixel adapter: A graph-based post-processing approach for scene text image super-resolution

    Wenyu Zhang, Xin Deng, Baojun Jia, Xingtong Yu, Yi- fan Chen, Jin Ma, Qing Ding, and Xinming Zhang. Pixel adapter: A graph-based post-processing approach for scene text image super-resolution. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2168–2179,

  81. [89]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 6, 7, 1

    2024 arXiv

  82. [90]

    Temporal action detection with structured segment networks

    Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xi- aoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE international conference on computer vision , pages 2914– 2923, 2017. 2

    2017

  83. [91]

    Multi-modal in-context learning makes an ego-evolving scene text recognizer

    Zhen Zhao, Jingqun Tang, Chunhui Lin, Binghong Wu, Can Huang, Hao Liu, Xin Tan, Zhizhong Zhang, and Yuan Xie. Multi-modal in-context learning makes an ego-evolving scene text recognizer. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2024

  84. [92]

    Mine- dreamer: Learning to follow instructions via chain-of- imagination for simulated-world control

    Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Mine- dreamer: Learning to follow instructions via chain-of- imagination for simulated-world control. arXiv preprint arXiv:2403.12037, 2024. 3 Number it: Temporal Grounding Video...

    2024 arXiv

  85. [93]

    Moment Retrieval In the training-free (NumPro) setting, we extract frames from videos at 1 FPS, with each frame resized to a reso- lution of 336 × 336

    Experimental Details 6.1. Moment Retrieval In the training-free (NumPro) setting, we extract frames from videos at 1 FPS, with each frame resized to a reso- lution of 336 × 336. In the fine-tuned (NumPro-FT) setting, frames are extracted at 0.5 FPS during both the training and...

  86. [94]

    from frame 000 to frame 200

    Hallucination in Vid-LLMs for VTG 7.1. General Vid-LLMs Qwen2-VL-7B. Figure 8a shows the results of Qwen2-VL- 7B [66] suffer from severe hallucinations. For instance, the model generates responses like “from frame 000 to frame 200” even when the input video contains only 19 fr...

  87. [95]

    from 2 to

    Additional Attention Analysis We present additional attention analysis results in Figure 10. In the examples on the left, the model produces incorrect or incomplete outputs, such as “from 2 to .”. On the right, the examples display severe hallucinations, with outputs ex- tendi...

  88. [96]

    We use 1FPS as the sampling rate, and adopt a design of red color, font size 40, and bottom right positioning for the number prompt

    Additional Video Benchmark Results We conducted experiments on additional video ques- tion answering (QA) benchmarks, MVBench [37] and VideoMME [16], as summarized in Table 8. We use 1FPS as the sampling rate, and adopt a design of red color, font size 40, and bottom right pos...

  89. [97]

    The results show that our method generalizes well across various General Vid-LLMs, achieving notable improvements in both mAP and HIT@1 metrics

    Ablation Results on Highlight Detection We present additional ablation results on the QVHighlights dataset, as shown in Table 5. The results show that our method generalizes well across various General Vid-LLMs, achieving notable improvements in both mAP and HIT@1 metrics. Spe...

  90. [98]

    While a font size of 60 achieves better Number Accu- racy than a size of 40 (Figure 5 of the main paper), it reduces Caption Accuracy and introduces more outliers

    Ablation Results on NumPro-FT Designs In this section, we present the ablation results for NumPro- FT. While a font size of 60 achieves better Number Accu- racy than a size of 40 (Figure 5 of the main paper), it reduces Caption Accuracy and introduces more outliers. Table 6 fu...

  91. [99]

    10.5s”) may introduce decimals, which can increase parsing complexity for Vid-LLMs. In Tabel 9, we compare minute-level temporal annotations (e.g., “01:10

    NumPro with Accurate Timestamps In the main paper, we choose frame numbers because they serve as the smallest discrete units of a video and can be di- rectly mapped to precise timestamps using the frame sam- pling rate. In this section, we compare the performance by directly o...

  92. [100]

    Dialogue Figure 11 illustrates a real-world application of our NumPro method within the Qwen2-VL-7B model, highlighting its ability to handle complex video-based dialogue tasks

    Additional Visualization Cases 13.1. Dialogue Figure 11 illustrates a real-world application of our NumPro method within the Qwen2-VL-7B model, highlighting its ability to handle complex video-based dialogue tasks. Compared to the VTG-specific models [10, 41], NumPro equipped ...

  93. [101]

    Expanding evaluation to include longer videos, such as hour-long recordings, is essential for test- ing the scalability and generalizability of our approach

    Limitations While NumPro and NumPro-FT have proven effective across multiple models and datasets, significantly surpass- ing previous state-of-the-art models, there are still some limitations: • Limited Dataset Scope: Current datasets for video tempo- ral grounding (VTG) tasks...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.