Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CaReBench introduces 1,000 videos with human-annotated captions manually split into spatial and temporal parts so that video-language models can be tested separately on static-object and dynamic-action understanding.

desk verdict A genuinely useful benchmark idea undermined by unvalidated spatial/temporal separation and a metric equation error—worth a major revision, not a desk reject. read the letter →

arxiv 2501.00513 v2 pith:GQYHSBEQ submitted 2024-12-31 cs.CV cs.IRcs.LG

Yifan Xu , Xinhao Li , Yichun Yang , Desen Meng , Rui Huang , Limin Wang This is my paper · ORCID
classification cs.CVcs.IRcs.LG
keywords videocaptioningretrievalfine-grainedbenchmarkspatiotemporalbiasspatial-temporalcaptionseparationmultimodallargelanguagemodelcontrastivelearningmetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CaReBench is a testing benchmark of 1,000 videos, each with a long human-written caption of 150 to 300 words organized hierarchically into summary, object, action, and miscellaneous descriptions. Its central design move is to split every caption by hand into a spatial description covering objects, attributes, layout, and background, and a temporal description covering actions, their order, camera movement, and subtitle changes. Because videos within each subcategory were chosen to look similar, the separated descriptions are meant to be hard enough to discriminate videos on their own. On top of the split captions the paper defines ReBias, a retrieval-bias score based on the ratio of temporal to spatial recall, and CapST, a captioning metric that uses an LLM to extract objects and events and then natural-language inference to compare them. The paper argues these tools let researchers tell whether a model's video-language failure comes from static-object understanding or from dynamic-action understanding, and that existing benchmarks and metrics cannot do this.

What carries the argument

The load-bearing object is the manually separated spatial and temporal caption pair for each video, where spatial captions keep objects, attributes, positions, layout, and background, and temporal captions keep actions, event order, camera movements, and subtitle changes while static references are removed. The benchmark's metrics are built directly on this separation: ReBias is $B=\left|1-\bar R_{\text{temporal}}/\bar R_{\text{spatial}}\right|$ using retrieval recall averaged over R@1, R@5, and R@10, and CapST uses an LLM to extract objects from spatial captions and events from temporal captions and then computes precision and recall through natural-language-inference entailment. The proposed baseline CARE carries the argument by treating captioning and retrieval as one map $\phi:\mathbb{R}^{T\times H\times W\times C}\to\mathbb{R}^{D}$ from pixels to either vocabulary space or embedding space, achieved by two-stage supervised fine-tuning of a 7B video-language model.

What would settle it

Have a second team of annotators re-perform the spatial/temporal split on a sample of the 1,000 captions without seeing the originals, then measure agreement by whether a held-out video can be identified from each re-split caption and whether the two teams' ReBias scores match; if the splits are not reproducible, or if the spatial-only and temporal-only captions fail to discriminate videos within a subcategory, the ReBias and CapST numbers would measure annotation noise rather than model bias.

Watch

Extended reading notes

Core claim

The paper's claim is that a benchmark whose ground-truth captions are manually separated into spatial and temporal components can independently quantify the spatial and temporal abilities of video-language models, and that the same separation can drive a unified model that retrieves and captions videos. Using 1,000 videos from FineAction, chosen so videos within a subcategory are visually and behaviorally similar, CaReBench asks annotators first to write detailed hierarchical descriptions and then to remove action-related text from object descriptions and static references from action descriptions. The resulting ReBias metric reports $B=\left|1-\bar R_{\text{temporal}}/\bar R_{\text{spatial}}\right|$, the gap between average recall on temporal and spatial retrieval, while CapST extracts objects and events from spatial and temporal captions and scores them by entailment. The paper's experiments show that every tested model retrieves spatial captions better than temporal captions, indicating a systematic bias toward static objects, and that a 7B multimodal LLM fine-tuned in two stages, first on fine-grained video-caption pairs and then with text-only contrastive learning on NLI data, can beat specialized CLIP-based retrieval models and specialist captioning MLLMs on the benchmark's fine-grained tasks.

Load-bearing premise

The benchmark's usefulness depends on human annotators being able to split each caption into a pure spatial part and a pure temporal part that are each, on their own, enough to tell similar videos apart, and the paper does not report inter-annotator agreement or a quality check on that separation.

Editorial extensions

If this is right

  • ReBias and CapST give a common way to report whether a model is skewed toward static objects or dynamic actions, so future video-language work can target the weaker side instead of averaging the two.
  • Because all tested models show a spatial advantage in retrieval, the results imply that temporal modeling is currently the larger bottleneck in video-language understanding.
  • The two-stage CARE recipe suggests that retrieval and captioning do not need separate architectures; a single multimodal LLM can generate detailed descriptions and extract retrieval embeddings, and each stage measurably helps the other.
  • The benchmark's long captions, averaging about 228 words, stress-test models in a way short-caption benchmarks cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a natural next test is whether training specifically to reduce ReBias, for example with temporal-contrastive losses, also improves temporal action grounding and video question answering, something the paper does not do.
  • Beyond the paper, the spatial/temporal separation scheme could be transferred to image-text or embodied datasets where static layout and dynamic state need to be disentangled.
  • Beyond the paper, because the paper reports no inter-annotator agreement on the separation step, an external audit of caption splits would tell whether the bias numbers reflect model limitations or annotation style.
  • Beyond the paper, CaReBench could serve as a diagnostic for comparing frame sampling strategies or video tokenizers on temporal retrieval, since the temporal captions isolate motion sensitivity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CaReBench, a benchmark of 1,000 videos from FineAction with detailed human-annotated captions structured hierarchically into summary, object, action, and misc parts, and manually separated into spatial and temporal descriptions. It proposes two metrics: ReBias, which measures the imbalance between spatial and temporal retrieval recall, and CapST, an NLI-based captioning metric that computes F1 over extracted objects and events. It also presents CaRe, a Qwen2-VL-based baseline trained with two-stage SFT to handle both video captioning and retrieval. Experiments report CaRe's performance on CaReBench and on traditional benchmarks MSR-VTT, MSVD, and DiDeMo.

Significance. If the spatial/temporal separation is valid, CaReBench would be a valuable resource for diagnosing whether video-language models fail on static or dynamic understanding. The two-metric design is a reasonable step toward finer-grained evaluation, and the CaRe baseline demonstrates a plausible path to unifying retrieval and captioning in one MLLM. The paper includes experiments on established benchmarks, which help situate the results. However, the central validity claim is not yet supported, and the CapST metric as written is internally inconsistent.

major comments (4)
  1. [Section 3.4.2, Eqs. (2)-(3)] The definitions of recall and precision in CapST are internally inconsistent. Eq. (3) divides by N(Dgt) rather than N(Dpred), and the entailment directions labeled recall and precision appear swapped. As written, R counts predicted elements entailed by the ground truth (which is precision) divided by the ground-truth count (wrong denominator), while P counts ground-truth elements entailed by the prediction (which is recall) divided again by the ground-truth count. This makes the F1 values in Table 2 irreproducible from the stated formulas. Please correct the definitions to the standard forms (recall = |Egt entailed by Dpred| / |Egt|, precision = |Epred entailed by Dgt| / |Epred|) and recompute the reported scores.
  2. [Section 3.2.2 and Appendix C] The central claim that the spatial and temporal captions are pure is not validated. Section 3.2.2 states that annotators 'eliminate static references' from action descriptions, but the examples in Appendix C contain explicit static scene references in the temporal annotations (e.g., 'in an amusement park' and 'on the fencing arena'), and Figure 4 shows 'in the center lane of the pool' in the temporal description. No inter-annotator agreement, expert-audit statistics, or discriminativeness checks are reported. Without such validation, ReBias and CapST may measure annotation artifacts rather than genuine spatiotemporal bias. Please provide a reliability study (e.g., Cohen's kappa on a sample, or a check that temporal captions alone can discriminate videos within a subcategory) and either revise the temporal annotations or the purity claims.
  3. [Section 5.2] CapST uses DeepSeek-V3 as an LLM judge, but no human validation of the judge's element extraction or NLI decisions is reported. The captioning scores in Table 2 therefore rest on an unverified proxy. Please include a small human agreement study (e.g., correlation with human judgments on a subset of predictions, or a qualitative error analysis) to establish that the metric behaves as intended.
  4. [Section 5 and Tables 2-5] All results are point estimates without confidence intervals or significance tests. On a 1,000-video benchmark, differences like the General Retrieval R@1 gap between CaRe (77.0) and Qwen2-VL (76.6) in Table 3 may be within sampling noise. Please report bootstrap confidence intervals or significance tests for the main comparisons, particularly for the ReBias and CapST scores on which the spatiotemporal-bias conclusions are based.
minor comments (4)
  1. [Throughout] There are several typographical errors: 'perfomance' in Section 3.4, 'rdaptation' in Section 5.4, 'Retreival' in Figure 2, and 'Comparision' in Figure 1.
  2. [Tables 6 and 7] The Qwen2-VL rows in Tables 6 and 7 cite reference [28] (Tarsier) instead of [29] (Qwen2-VL); the citations should be corrected.
  3. [Section 3.4.1] ReBias averages R@1, R@5, and R@10, which have different scales and are not directly comparable. Consider reporting per-K values or normalizing the recalls before averaging, and justify the chosen aggregation.
  4. [Section 5] The paper describes the captioning and retrieval results as 'zero-shot', but CaRe is fine-tuned on video-text data in Stage-I. Please clarify what 'zero-shot' means in this context, since the baseline is not zero-shot.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ReBias, CapST, and the CARE baseline are evaluated against external data and independent judges, with no fitted parameter renamed as a prediction.

full rationale

I walked the paper's claimed derivation chain: CaReBench annotations feed ReBias and CapST, and CARE is trained and evaluated against them. None of these steps reduces to its own inputs by construction. ReBias (Eq. 1) is a ratio of two separately measured recalls on manually separated caption sets; the separation is an annotation property, not a parameter fitted to model outputs. CapST (Eqs. 2-3) uses DeepSeek-V3 as an external LLM judge and NLI entailment, and the paper reports zero-shot evaluation on models not trained on CaReBench. CARE's two-stage SFT uses public data (LLaVA-Video-178k, Tarsier recaptions, and the SimCSE NLI dataset), not the benchmark's own captions or retrieval targets, so its reported performance is not forced by a fitted input. The paper's self-citations (e.g., FineAction as a video source, InternVideo2 as a comparison) are contextual rather than load-bearing for the central spatiotemporal-bias claim. The main limitation is validity rather than circularity: Section 3.2.2 asserts that temporal descriptions 'exclude any obvious static object descriptions,' but the paper reports no inter-annotator agreement or discriminativeness audit, and the Appendix C temporal annotations retain static references such as 'in an amusement park' and 'on the fencing arena.' This could make ReBias and CapST reflect annotation artifacts, but it is a measurement-validity concern, not a circular derivation. The conclusion's admission that CARE 'doesn’t address problems about VLMs’ bias' is also a limitation statement, not evidence of circularity. I therefore find no circular step warranting a nonzero score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central contributions rest on the benchmark annotations and the LLM-judged metric; no new physical or mathematical entities are postulated.

free parameters (1)
  • temperature tau in contrastive loss (Eq. 4) = not reported
    Chosen by hand with no sensitivity analysis; it scales the cosine similarities that determine retrieval rankings, so the headline R@1 numbers depend on it.
assumptions (4)
  • domain assumption FineAction videos are a representative and sufficiently challenging sample for fine-grained video understanding.
    The benchmark is built entirely from 1,000 manually filtered FineAction clips; if this distribution is not representative, the measured biases do not generalize.
  • domain assumption Human annotations are accurate ground truth with no hallucinated details.
    No inter-annotator agreement or error rate is reported, yet all captioning and CapST scores treat the human captions as correct.
  • domain assumption DeepSeek-V3 NLI entailment judgments reliably capture whether extracted elements are supported.
    CapST uses an LLM judge without validating against human NLI judgments; any systematic LLM bias transfers into the metric.
  • domain assumption Manual separation into spatial and temporal captions is consistent and each part is independently discriminative.
    This is the unique feature of the benchmark; Section 3.2.2 asks for it but provides no consistency check.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval." pith.science (2026). https://pith.science/paper/GQYHSBEQ

@misc{pith2026250100513,
  author       = {Pith},
  title        = {Pith review of: CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQYHSBEQ}},
  note         = {Machine review of arXiv:2501.00513}
}
read the original abstract

Video understanding, including video captioning and retrieval, is still a great challenge for video-language models (VLMs). The existing video retrieval and caption benchmarks only include short descriptions, limits their ability of detailed video understanding evaluation. To address this problem, we present CaReBench, a testing benchmark for fine-grained video captioning and retrieval with 1,000 high-quality pairs of videos and human-annotated detailed captions. Uniquely, it provides manually separated spatial annotations and temporal annotations for each video. Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks, respectively. These metrics enable a comprehensive investigation into the spatial and temporal biases inherent in VLMs. In addition, to handle both video retrieval and video captioning tasks in a unified framework, we develop a simple baseline based on a Multimodal Language Model (MLLM). By implementing a two-stage Supervised Fine-Tuning (SFT), we fully unlock the potential of MLLM, enabling it not only to generate detailed video descriptions but also to extract video features. Surprisingly, experimental results demonstrate that, compared to the CLIP-based models designed for retrieval and the popular MLLMs skilled in video captioning, our baseline shows competitive performance in both fine-grained video retrieval and video detailed captioning.

Figures

Figures reproduced from arXiv: 2501.00513 by the authors.

Figure 1
Figure 1. Comparision of captions between MSR-VTT [33], GPT-4o generated data [7] and CAREBENCH. The caption in the upper left corner is from MSR-VTT [33]. It only contains short-text coarse descriptions. The annotation located in the lower left corner is generated by GPT-4o sourced from ShareGPT-4o [7]. It has some coarse-grained, uncertain and wrong descriptions. The fine-grained caption on the right is selected from CAREBE… view at source ↗
Figure 2
Figure 2. Comparison on the CAREBENCH performance of CLIP-based retrieval models, MLLM captioning models and our unifed model. The results on MLLMs are reported on their public version without contrastive training. The CLIP-based retrieval model has achieved excellent performance in video retrieval tasks, but it lacks the ability to describe videos. On the other hand, MLLM models are capable of describing videos in detail, bu… view at source ↗
Figure 3
Figure 3. Statistics of CAREBENCH. Most videos range from 5-20 seconds and most captions fall between 150 and 300 words in length. inate similar videos. We manually select 1,000 videos from FineAction [21] with 10-20 videos in each subcategory. Videos are filtered out that (1) are not clear enough, (2) contain little actions and movements, and (3) contain vastly different scenes and actions which are easy for VLMs to discrimi… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An overview of the annotation pipeline. In Stage-I, workers are asked to describe videos hierarchically in detail. In Stage-II, workers need to separate spatial descriptions with temporal descriptions. can be divided into four parts: a general overview, a spatial descr…
Figure 5
Figure 5. Figure 5: The training recipe of CARE. In the first stage, we align CARE outputs to a fine-grained text space, enabling it to describe videos in detail. In the second stage, a contrastive learning method is applied to get features from the inputs. The output space of CARE shifts…
Figure 6
Figure 6. Figure 6: Top 50 tokens decoded from the output embeddings of Qwen2-VL and CARE. Qwen2-VL is the baseline model of CARE without any SFT. Compared to Qwen2-VL, two-stage SFT makes the semantic components of CARE embedding much more related to the input video featuring a chef is c…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GRPO post-training with structured thinking and dual think/caption rewards improves Qwen2-VL-7B video captioning over the base model and SFT on DREAM-1K, VDC, and CAREBENCH.

  2. Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.

  3. PercepCap: Video Captioner with Structured Spatio-Temporal Perception

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Explicitly generating object trajectories and event timestamps before the final caption improves detailed video captioning on multiple benchmarks, though the improvement is largely driven by an external perception oracle.

Reference graph

Works this paper leans on

41 extracted references · 30 canonical work pages · cited by 3 Pith papers

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    work page 2020

  2. [2]

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D. Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. CoRR, abs/2410.03051, 2024. 2, 3

    arXiv

  3. [3]

    Chen and William B

    David L. Chen and William B. Dolan. Collecting highly par- allel data for paraphrase evaluation. In ACL, pages 190–200. The Association for Computer Linguistics, 2011. 2, 3, 6, 1

    work page 2011

  4. [4]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. CoRR, abs/2312.14238, 2023. 4, 8, 9, 2

    arXiv 2023

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- 10 wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 8

    arXiv 2024

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    work page 2023

  7. [7]

    Sharegpt-4o: Comprehensive mul- timodal annotations with gpt-4o, 2024

    Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive mul- timodal annotations with gpt-4o, 2024. 1

    work page 2024

  8. [8]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    arXiv 2024

Show all 41 references
  1. [9]

    BERT: pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguis- tics, 2019. 4

    2019

  2. [10]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever compreh...

    2024 arXiv

  3. [11]

    Simcse: Sim- ple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Sim- ple contrastive learning of sentence embeddings. In EMNLP (1), pages 6894–6910. Association for Computational Lin- guistics, 2021. 7

    2021

  4. [12]

    Imagebind one embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In CVPR, pages 15180–15190. IEEE, 2023. 3

    2023

  5. [13]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970. IEEE Computer Society, 2015. 3

    2015

  6. [14]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. Localizing mo- ments in video with natural language. In ICCV, pages 5804–

  7. [15]

    Scaling sentence embeddings with large language models

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. In EMNLP (Findings), pages 3182–3196. Association for Computational Linguistics, 2024. 7

    2024

  8. [16]

    E5-V: universal embeddings with multi- modal large language models

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: universal embeddings with multi- modal large language models. CoRR, abs/2407.12580, 2024. 2, 4, 7

    2024 arXiv

  9. [17]

    Videochat: Chat-centric video understanding

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.06355,

    arXiv

  10. [18]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pages 19891–19903. IEEE, 2023. 3

    2023

  11. [19]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In CVPR, pages 22195–22206. IEEE, 2024. 4

    2024

  12. [20]

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Ye- ung, and James Y . Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022. 4

    2022

  13. [21]

    Fineaction: A fine-grained video dataset for temporal action localization

    Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. IEEE Trans. Image Process. , 31:6937–6950,

  14. [22]

    Clip4clip: An empirical study 11 of CLIP for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study 11 of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304, 2022. 2, 3

    2022

  15. [23]

    X-CLIP: end-to-end multi-grained con- trastive learning for video-text retrieval

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-CLIP: end-to-end multi-grained con- trastive learning for video-text retrieval. InACM Multimedia, pages 638–647. ACM, 2022. 2, 3

    2022

  16. [24]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,

    arXiv

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748–...

    2021

  18. [26]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In CVPR, pages 4566–4575. IEEE Computer Society,

  19. [27]

    GIT: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Trans. Mach. Learn. Res., 2022, 2022. 2, 3

    2022

  20. [28]

    Tarsier: Recipes for training and evaluating large video description models

    Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models. CoRR, abs/2407.00634, 2024. 2, 3, 6, 7, 8, 9

    2024 arXiv

  21. [29]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  22. [30]

    Intern- video2: Scaling foundation models for multimodal video un- derstanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Intern- video2: Scaling foundation models for multimodal vide...

    2024

  23. [31]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learn- ers. In ICLR. OpenReview.net, 2022. 4

    2022

  24. [32]

    mplug-2: A modularized multi-modal foundation model across text, image and video

    Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jin- gren Zhou. mplug-2: A modularized multi-modal foundation model across text, image and video. In ICML, pages 3872...

    2023

  25. [33]

    MSR-VTT: A large video description dataset for bridging video and lan- guage

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and lan- guage. In CVPR, pages 5288–5296. IEEE Computer Society,

  26. [34]

    Vript: A video is worth thousands of words

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems, 37:57240–57261, 2024. 2

    2024

  27. [35]

    Minicpm-v: A GPT-4V level MLLM on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. M...

    2024 arXiv

  28. [36]

    Long-clip: Unlocking the long-text capability of CLIP

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of CLIP. In ECCV (51), pages 310–325. Springer, 2024. 2, 3, 8, 9

    2024

  29. [37]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 4, 7, 8, 9, 2

    2024

  30. [38]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 2, 7

    2024

  31. [39]

    VISTA: visualized text embedding for universal multi-modal retrieval

    Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yong- ping Xiong. VISTA: visualized text embedding for universal multi-modal retrieval. In ACL (1), pages 3185–3200. Asso- ciation for Computational Linguistics, 2024. 2, 3, 4

    2024

  32. [40]

    NOW I’VE SEEN EVERY- THING

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zong- wei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. In ICLR...

    2024

  33. [5813]

    2, 3, 6, 1

    IEEE Computer Society, 2017. 2, 3, 6, 1

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.