Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Progress-Aware Video Frame Captioning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 7B captioning model called ProgressCaptioner can describe how an action progresses frame by frame, and it reportedly beats the much larger GPT-4o and Gemini-1.5-Pro on the new FrameCapEval benchmark.

desk verdict A genuinely new task and a solid empirical pipeline, but the benchmark critics overlap with the training critics enough that the headline margins need independent human validation before they are taken at face value. read the letter →

arxiv 2412.02071 v2 pith:NGYBVWWR submitted 2024-12-03 cs.CV

Zihui Xue , Joungbin An , Xitong Yang , Kristen Grauman This is my paper · ORCID
classification cs.CV
keywords videocaptioningframe-levelcaptionsactionprogressiontemporalhallucinationpseudo-labelingdirectpreferenceoptimizationvision-languagemodelsFrameCapEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video captioning usually yields one description for a whole clip, while image captioning treats each frame alone. This paper stakes out a middle ground: generate a caption for every frame that both describes what is visible and states how the action has progressed since the previous frame. The authors argue that existing vision-language models fail at this—they either produce near-identical captions for subtly different frames or hallucinate progress that the frames do not show. To close the gap, they introduce ProgressCaptioner, a 7B model trained with a two-stage pseudo-labeling pipeline, and claim it outperforms much larger models such as GPT-4o and Gemini-1.5-Pro on a new benchmark, FrameCapEval. If correct, the approach would provide a scalable source of temporally fine-grained captions for coaching, how-to videos, keyframe selection, and video understanding.

What carries the argument

The load-bearing mechanism is frame-pair captioning embedded in a two-stage training loop, with automatic critics standing in for human annotation. Stage I trains the model to caption pairs of adjacent frames, so the caption for the second frame must identify what changed and what did not; Stage II extends to full sequences by sliding a two-frame window over the video while also mixing in captions from other VLMs. The two critics are progression detection, where an LLM (Llama-3.1-70B-Instruct) decides whether a caption pair reflects a visible physical change using majority voting across multiple LLM judgments, and caption matching, where a VLM must match each caption to its correct frame among hard negatives plus an unsure option. Captions that pass both critics become supervised fine-tuning data; those that fail become preference pairs for direct preference optimization (DPO), which steers the model away from temporal hallucination. The same critics then serve as the benchmark metrics, with Gemini-1.5-Pro as evaluation VLM.

What would settle it

Replace the two critics with human raters on a sample of FrameCapEval (for example, asking people to judge per-frame factual correctness and whether each caption sequence reflects true action progression, without using Llama-3.1 or a VLM as judge) and check whether ProgressCaptioner still ranks above GPT-4o and Gemini-1.5-Pro; a ranking flip or a large drop in ProgressCaptioner's margin would show the benchmark gains are partly an artifact of the critic loop.

Watch

Extended reading notes

Core claim

ProgressCaptioner establishes that a compact model, initialized from a general vision-language model and trained only on automatically filtered pseudo-labels, can produce frame-level captions that are both specific to each frame and coherently track the progression of an action over time. On FrameCapEval, built from HowToChange, COIN, Penn Action, and Kinetics, the model reports 1.8x–2.7x improvements in caption matching and progression detection accuracy over leading open-source VLMs of similar size, and it out-scores the much larger Gemini-1.5-Pro and GPT-4o on both automatic metrics. In a user study with 15 participants, ProgressCaptioner receives the highest top-1 selection rate, about 2x–3.6x better than comparably sized open-source competitors, although proprietary models remain competitive on the more forgiving top-2 metric. The captions also drive practical gains: they feed keyframe selection that improves action recognition (+1.7% top-1 on Kinetics for both SlowFast and X3D) and boost zero-shot frame classification and video QA (+3.4% over the previous leader on NExT-QA ATP-Hard).

Load-bearing premise

The automatic critics used to filter training data and to score the benchmark are assumed to be valid substitutes for human judgment about whether a caption accurately and specifically describes its frame.

Editorial extensions

If this is right

  • Progress-aware frame captions can serve as a textual intermediate representation for keyframe selection, enabling an LLM to pick frames that mark distinct stages of an action rather than near-duplicates.
  • Feeding these captions to an LLM improves zero-shot frame-wise action classification and temporally challenging video QA, outperforming prior zero-shot approaches on NExT-QA ATP-Hard.
  • The same automatic progression-detection and caption-matching critics can filter pseudo-labeled data at scale, reducing reliance on human annotation for temporally fine-grained captioning datasets.
  • Because ProgressCaptioner accepts 2 to T input frames, users can trade off local frame-window detail against global event context by adjusting the temporal context.
  • The model generalizes to datasets unseen in training (Penn Action and Kinetics), indicating that the learned progress-awareness transfers beyond the source instructional videos.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper uses the same critic mechanisms for data selection and evaluation, which raises a possible circularity; a useful test would be training a variant with human-filtered data and measuring how much of the benchmark gain is attributable to the critic loop itself.
  • The frame-pair atomic unit suggests the approach will help most where adjacent frames show small but real state changes; on videos with large visual jumps or near-static segments, the critics' binary change judgment may be less informative.
  • A natural extension is to apply the captions as auxiliary input to video retrieval or dense video captioning systems; if frame-level progress captions carry the same discriminative signal shown here, retrieval and summarization quality should also improve.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces progress-aware video frame captioning, a task that requires per-frame captions that preserve temporal progression. It presents ProgressCaptioner, a 7B VLM trained in two stages: first on frame pairs and then on frame sequences, using pseudo-labels generated by an ensemble of VLMs and filtered by two automatic critics (progression detection and caption matching), followed by SFT and DPO. The authors also introduce the FrameCap dataset and the FrameCapEval benchmark, and report that ProgressCaptioner outperforms open-source VLMs and proprietary models such as GPT-4o and Gemini-1.5-Pro on caption matching and progression detection, with supporting qualitative results, a user study, and downstream applications in keyframe selection and video understanding.

Significance. If the reported results are reliable, the paper makes a useful contribution by defining a new task, providing a scalable pseudo-labeling pipeline, and releasing what could become a standard benchmark for frame-level caption evaluation. The manual verification of benchmark videos, the two-stage training design, and the downstream application results (keyframe selection, action recognition, video QA) are concrete strengths. The main risk is that the evaluation metrics are tightly coupled to the training critics, so the headline gains may partly reflect alignment with those critics rather than independently validated caption quality. Because the authors explicitly acknowledge limitations (noise in automatic filtering, the failure case in Fig. 20), the manuscript is honest, but the central empirical claim needs stronger independent validation before the results can be fully accepted.

major comments (4)
  1. [Sec. 3.2 vs. Sec. 4.1; Supp. Sec. 1 and 2.1] The automatic evaluation metrics in FrameCapEval are the same tasks and closely related model instances as the critics used to filter pseudo-labeled training data and to construct DPO preferences. Specifically, progression detection in both the training pipeline and the benchmark uses Llama-3.1-70B-Instruct, and caption matching is the same multi-choice hard-negative task, executed by VILA during data filtering and by Gemini-1.5-Pro during evaluation. Since ProgressCaptioner is SFT/DPO-trained to maximize acceptance by these critics, the benchmark partially measures alignment with the training critics. This is not worst-case circularity because the evaluation is on held-out videos and the evaluator VLM differs from the filtering VLM, but it is a substantial coupling. The abstract's claim that ProgressCaptioner 'greatly outperforms' GPT-4o and Gemini-1.5-Pro rests on these metrics. Please provide independent validation, for example: human annotations of progression labels and caption-frame correspondence on a subset of FrameCapEval, a report of metric-human agreement, and a comparison of all models on the human-validated subset. Alternatively, add a reference-based evaluation with human-written frame captions.
  2. [Sec. 4.1, Table 1; Supp. Table 4] The benchmark's caption-matching test sets are small (e.g., 102 HowToChange videos, 52 Kinetics videos, and 47 Penn Action videos), and the numbers in Table 1 are point estimates with no confidence intervals or significance tests. The user study (15 participants, 85 sequences) similarly reports no uncertainty. Given these small test sets, the 1.8x-2.7x improvement claims need bootstrap confidence intervals or a significance test, and the user study should report per-participant variance or inter-annotator agreement. Without these, it is difficult to know whether the gap over GPT-4o (e.g., 37.3 vs. 32.4 on HTC caption matching) is reliable.
  3. [Sec. 4.1, Fig. 7; Supp. Sec. 2.1] The user study results are more nuanced than the text suggests. Under the top-1 selection metric ProgressCaptioner leads, but under the top-2 metric GPT-4o scores 50.3% and ProgressCaptioner 47.3%, meaning the model is not the most preferred when participants can select two captions. The paper acknowledges this in the text, but the abstract and conclusion state unconditional superiority. Please qualify the superiority claims according to the two user-study metrics and report whether the top-1 advantage is statistically significant (e.g., a Wilcoxon signed-rank test over participants).
  4. [Supp. Limitations and Fig. 20] The paper's own limitation section reports a failure case where ProgressCaptioner produces identical captions for four frames that contain fine-grained spatial differences. This raises a validity question for the benchmark: if the caption-matching evaluator (Gemini-1.5-Pro) also fails to perceive such differences, the metric would not penalize this error, and the benchmark could overstate the model's temporal precision. Please analyze whether this failure case would pass the automatic caption-matching and progression-detection evaluations, and discuss the metric's sensitivity to such fine-grained failures.
minor comments (5)
  1. [Supp. Sec. 2.1] There is an unresolved reference: 'For results in Section ??, we employ a 2-frame sliding window' should point to the appropriate section in the main text.
  2. [Sec. 4.1] The manual verification of benchmark videos is a strength, but the paper should state how many videos were excluded during verification and describe the verification protocol (e.g., number of annotators and agreement measure).
  3. [Fig. 7] The user study bar chart would be more informative with error bars or a per-participant scatter, given the small number of participants.
  4. [Abstract and Sec. 5] The phrase 'significantly surpasses' appears without supporting statistical tests; consider softening to 'outperforms' or adding the required significance evidence.
  5. [Sec. 3.2] The name 'ProgressCaptioner' is used for both the Stage-I model and the final two-stage model; please clarify the terminology when the stage-I model is first introduced.

Circularity Check

2 steps flagged · score 6.0 of 10

FrameCapEval's two automatic metrics are the same critic tasks that filter pseudo-labels and build DPO preferences; progression detection even reuses the identical Llama-3.1-70B judge, so the 1.8x-2.7x headline gains partly measure alignment with the training critics.

  1. fitted input called prediction [Sec. 3.2 (data filtering & DPO) vs Sec. 4.1 (evaluation metric); Supp. Sec. 1.1]
    "Here, we employ progression detection and caption matching to automatically construct preference data ˆc+ and ˆc−, eliminating the reliance on manual labeling. ... The progression detection uses LLAMA-3.1-70B-Instruct [14], and for caption matching, we use VILA [40], chosen for its open-source availability and strong performance. ... We employ the automatic evaluation tasks of progression detection and caption matching (Sec. 3.2), reporting accuracy with Llama-3.1-70B-Instruct [14] as the evaluation LLM and Gemini-1.5-Pro [53] as the evaluation VLM."

    The model is explicitly trained to satisfy these critics: SFT uses caption pairs that passed progression detection, and DPO uses preference pairs built by the same progression-detection and caption-matching evaluations. The evaluation then reuses the same judge for the headline 'Prog' metric: Llama-3.1-70B-Instruct both filters pseudo-labels during training and scores progression detection on FrameCapEval. Because the model is optimized (via SFT and DPO) to make this exact LLM approve its captions, the reported progression-detection accuracy is partly a measure of agreement with the training critic rather than an independent measurement of progress awareness.

  2. fitted input called prediction [Sec. 3.2 (caption matching filtering & DPO) vs Sec. 4.1 (caption matching metric); Supp. Sec. 1.1]
    "A caption pair is considered good if the evaluation VLM correctly identifies ˆc1 for v1 and ˆc2 for v2. Because the captions will all be topically related, this is essentially a matching task with 'hard negatives' that lets us automatically gauge the precision of the proposed captions for the target images. ... Given that Gemini-1.5-Pro API usage incurs a cost, we reserve it for evaluation while utilizing the cost-free VILA as the caption matching evaluation VLM during the pseudo labeling stage."

    The caption-matching task is used twice: first to filter pseudo-labeled caption pairs and sequences into SFT positives and DPO preference data, and second as the 'Cap' metric on FrameCapEval. The model is SFT/DPO-optimized to produce captions that a VLM can correctly match to frames, since only caption pairs/sequences that passed this VLM matching test were used as positive training data. The benchmark evaluates the same multiple-choice frame-to-caption matching task, albeit with Gemini-1.5-Pro instead of VILA. Swapping the specific VLM does not remove the coupling: the training objective is to maximize matchability by this class of VLM critics, so the caption-matching score measures success at the trained objective. The reliability check in Fig.

full rationale

The paper's central numeric claim is that ProgressCaptioner achieves 1.8x-2.7x improvements over leading VLMs on FrameCapEval. The derivation chain behind that claim is partially circular. The same automatic critic tasks used for pseudo-label filtering and DPO preference construction are reused as the two FrameCapEval metrics. Most directly, Llama-3.1-70B-Instruct judges progression detection both when building <v, c+, c-> pairs for training and when computing the headline 'Prog' accuracy, so the model is trained to satisfy that judge and then scored by that judge. The caption-matching metric is somewhat less tightly coupled because the training critic (VILA) and evaluation critic (Gemini-1.5-Pro) differ, but the task is identical and the model was optimized for matchability by VLM critics. The supplementary reliability check only swaps in other LLM/VLM evaluators (GPT-4o-mini, GPT-4o) of the same class, so it does not validate the metrics against an independent human ground truth. The paper itself concedes that such automatic filtering 'introduces noise—though less costly, it's not as reliable as human annotation.' The independent user study and the downstream application results (keyframe selection, action recognition, NExT-QA) provide non-circular evidence that the captions have practical utility, which prevents a score of 8 or 10. However, the specific automatic-metric improvements in Table 1 are substantially coupled to the training critics and should be interpreted as measuring critic alignment rather than fully independent progress-aware caption quality.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several hand-chosen thresholds and two domain assumptions about the sufficiency of 1FPS sampling and the validity of the automatic critics. No new physical or conceptual entities are introduced.

free parameters (3)
  • caption_matching_error_threshold = >50% incorrect answers marks a sequence as rejected
    Hand-chosen threshold in Stage II data preparation (Sec 3.2) that determines which caption sequences enter DPO preference data; it shapes the training distribution and is not derived from first principles.
  • frame_sequence_length_cap = 6 frames
    Set based on preliminary experiments showing degradation beyond 6 frames (Supp Sec 1.1); it defines the input window for training and evaluation and is a manual design choice.
  • majority_voting_rule = majority across LLM assessments and across K captioners
    Used in progression detection to establish consensus visual-change labels and to select M distinct frames (Sec 3.2); chosen by hand with no reported sensitivity analysis.
assumptions (3)
  • domain assumption Uniform 1FPS sampling of action clips yields frames that sufficiently expose action progression.
    Section 3.1 states frames are sampled at 1FPS; the whole task assumes that progress is visible at this temporal rate, and the benchmark explicitly excludes videos without progression.
  • domain assumption The automatic critics (Llama-3.1-70B for progression detection, VLM for caption matching) are valid proxies for human judgment of caption quality.
    Used in Sec 3.2 both to filter training data and in Sec 4.1 to score models; reliability is only checked by consistency across alternative evaluators (Fig 19) and a small 90-question subset for VLM selection, not against human annotations on the benchmark itself.
  • standard math Initializing from LLaVA-OV-7B transfers general VLM captioning ability to the frame-captioning task.
    Sec 3.2 Stage I Training; this is a standard transfer-learning assumption in current VLM practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Progress-Aware Video Frame Captioning." pith.science (2026). https://pith.science/paper/NGYBVWWR

@misc{pith2026241202071,
  author       = {Pith},
  title        = {Pith review of: Progress-Aware Video Frame Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGYBVWWR}},
  note         = {Machine review of arXiv:2412.02071}
}
read the original abstract

While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility.

Figures

Figures reproduced from arXiv: 2412.02071 by the authors.

Figure 1
Figure 1. We propose progress-aware video frame captioning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Use cases of video frame captioning: finer-grained cap [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Issues of existing VLMs in video frame captioning: (1) [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Captioning outcomes using Gemini-1.5-Pro [ [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Framework of ProgressCaptioner, designed in two stages. In Stage-I, we prepare frame pairs and generate corresponding caption [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparisons of ProgressCaptioner with SOTA VLMs on three action sequences. For sequences (a) and (b), only [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: User study results comparing ProgressCaptioner with [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: ProgressCaptioner facilitates keyframe selection and enriches the selected keyframes with progress-aware descriptions. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: On Kinetics test videos, ProgressCaptioner selects four [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Example of a frame pair (decided by progression de [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Example of a frame pair (decided by caption match [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Example of a frame sequence. Progression detection is first applied to each adjacent frame pair to determine the visual-change [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Qualitative comparisons of ProgressCaptioner with SOTA VLMs (I). Red text identifies inaccuracies in the generated captions, [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Qualitative comparisons of ProgressCaptioner with SOTA VLMs (II). Red text identifies inaccuracies in the generated captions, [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Qualitative comparisons of ProgressCaptioner with SOTA VLMs (III). Red text identifies inaccuracies in the generated captions, [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Captions produced by ProgressCaptioner and processed by an LLM enable us to automatically select representative frames that [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Comparison of our keyframe selection with V2Xum [ [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Caption matching (left) and progress detection (right) [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: One failure case of ProgressCaptioner, where it fails to [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.

  2. Vid2Coach: Transforming How-To Videos into Task Assistants

    cs.HC 2025-05 conditional novelty 6.0 of 10

    Vid2Coach converts how-to videos into a real-time, wearable task assistant that helps blind and low vision people cook with fewer errors.

Reference graph

Works this paper leans on

89 extracted references · 48 canonical work pages · cited by 2 Pith papers

  1. [1]

    A Review of Deep Learning for Video Captioning

    Moloud Abdar, Meenakshi Kollati, Swaraja Kura- parthi, Farhad Pourpanah, Daniel McDuff, Mohammad Ghavamzadeh, Shuicheng Yan, Abduallah Mohamed, Abbas Khosravi, Erik Cambria, et al. A review of deep learning for video captioning. arXiv preprint arXiv:2304.11431 , 2023. 1, 3

    work page Pith review arXiv 2023

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Nocaps: Novel object cap- tioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. Nocaps: Novel object cap- tioning at scale. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 8948–8957, 2019. 2

    2019

  4. [4]

    Vismin: Visual minimal-change understanding

    Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal-change understanding. arXiv preprint arXiv:2407.16772, 2024. 3

    arXiv 2024

  5. [5]

    Revisiting the “video” in video-language understanding

    Shyamal Buch, Crist ´obal Eyzaguirre, Adrien Gaidon, Jia- jun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “video” in video-language understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2917–2927, 2022. 8

    2022

  6. [6]

    Temporalbench: Benchmarking fine- grained temporal understanding for multimodal video mod- els

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine- grained temporal understanding for multimodal video mod- els. arXiv preprint arXiv:2410.10818, 2024. 2

    arXiv

  7. [7]

    Quo vadis, ac- tion recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, ac- tion recognition? a new model and the kinetics dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299–6308, 2017. 5, 8, 1, 2

    2017

  8. [8]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 2, 3

    arXiv

Show all 89 references
  1. [9]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. InProceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies, pages 190–200, 2011. 2, 3

    2011

  2. [10]

    Rextime: A benchmark suite for reasoning-across-time in videos

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen- Chun Chen, and Yu-Chiang Frank Wang. Rextime: A benchmark suite for reasoning-across-time in videos. arXiv preprint arXiv:2406.19392, 2024. 3

    2024 arXiv

  3. [11]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 2, 3, 4

    2024 arXiv

  4. [12]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In IEEE/CVF Conference on Com- puter V...

    2024

  5. [13]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2

    2015 arXiv

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  7. [15]

    Describing differences in image sets with natural language

    Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 24199–...

    2024

  8. [16]

    X3d: Expanding architectures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 203–213, 2020. 8

    2020

  9. [17]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6202–6211, 2019. 8

    2019

  10. [18]

    Agqa: A benchmark for compositional spatio-temporal reasoning

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Ma- neesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11287–11297, 2021. 3

    2021

  11. [19]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In IEEE/...

    2024

  12. [20]

    Autoad: Movie description in context

    Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad: Movie description in context. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18930–18940, 2023. 3

    2023

  13. [21]

    Autoad ii: The sequel-who, when, and what in movie audio description

    Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. Autoad ii: The sequel-who, when, and what in movie audio description. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 13645–13655, 2023

    2023

  14. [22]

    Autoad iii: The prequel-back to the pixels

    Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad iii: The prequel-back to the pixels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18164–18174, 2024. 3

    2024

  15. [23]

    A comprehensive survey of deep learn- ing for image captioning

    MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratud- din, and Hamid Laga. A comprehensive survey of deep learn- ing for image captioning. ACM Computing Surveys (CsUR), 51(6):1–36, 2019. 1

    2019

  16. [24]

    V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning

    Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning. arXiv preprint arXiv:2404.12353, 2024. 7, 8, 9

    2024

  17. [25]

    Visual storytelling

    Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Is- han Misra, Aishwarya Agrawal, Jacob Devlin, Ross Gir- shick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association f...

    2016

  18. [26]

    Video re- cap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Na- garajan, Lorenzo Torresani, and Gedas Bertasius. Video re- cap: Recursive captioning of hour-long videos. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18198–18208, 2024. 3

    2024

  19. [27]

    Learning to describe differences between pairs of similar images

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv:1808.10584, 2018. 3

    2018 arXiv

  20. [28]

    Img-diff: Contrastive data synthesis for multimodal large language models

    Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. Img-diff: Contrastive data synthesis for multimodal large language models. arXiv preprint arXiv:2408.04594, 2024. 3

    2024 arXiv

  21. [29]

    Scsampler: Sampling salient clips from video for efficient action recog- nition

    Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recog- nition. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6232–6242, 2019. 8

    2019

  22. [30]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 706–715, 2017. 3

    2017

  23. [31]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 6

    2024 arXiv

  24. [32]

    Revealing single frame bias for video-and-language learning

    Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428, 2022. 5, 1

    2022 arXiv

  25. [33]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 3, 5, 6, 7, 8, 1

    2024 arXiv

  26. [34]

    Wolf: Captioning everything with a world summariza- tion framework

    Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, et al. Wolf: Captioning everything with a world summariza- tion framework. arXiv preprint arXiv:2407.18908, 2024. 3, 5

    2024 arXiv

  27. [35]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 1

    2024 arXiv

  28. [36]

    Video storytelling: Textual summaries for events

    Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankan- halli. Video storytelling: Textual summaries for events. IEEE Transactions on Multimedia, 22(2):554–565, 2019. 3

    2019

  29. [37]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational Conference on Machine Learning (ICML), pages 19730–19742. PMLR, 2023. 1, 2, 3

    2023

  30. [38]

    Visual to text: Survey of image and video captioning.IEEE Transactions on Emerging Topics in Computational Intelligence , 3(4):297– 312, 2019

    Sheng Li, Zhiqiang Tao, Kang Li, and Yun Fu. Visual to text: Survey of image and video captioning.IEEE Transactions on Emerging Topics in Computational Intelligence , 3(4):297– 312, 2019. 1

    2019

  31. [39]

    Vitatecs: A diag- nostic dataset for temporal concept understanding of video- language models

    Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diag- nostic dataset for temporal concept understanding of video- language models. arXiv preprint arXiv:2311.17404 , 2023. 3

    2023 arXiv

  32. [40]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 26689–26699,

  33. [41]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024

    2024

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. 2, 3, 5

    2024

  35. [43]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 3

    2024 arXiv

  36. [44]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems (NeurIPS) , 36:46212–46244,

  37. [45]

    Verbs in action: Improving verb understanding in video-language models

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 15579–15591, 2023. 8

    2023

  38. [46]

    Narrativebridge: Enhancing video captioning with causal- temporal narrative

    Asmar Nadeem, Faegheh Sardari, Robert Dawes, Syed Sameed Husain, Adrian Hilton, and Armin Mustafa. Narrativebridge: Enhancing video captioning with causal- temporal narrative. arXiv preprint arXiv:2406.06499, 2024. 3

    2024 arXiv

  39. [47]

    Oscar: Object state cap- tioning and state change representation

    Nguyen Nguyen, Jing Bi, Ali V osoughi, Yapeng Tian, Pooyan Fazli, and Chenliang Xu. Oscar: Object state cap- tioning and state change representation. arXiv preprint arXiv:2402.17128, 2024. 3

    2024 arXiv

  40. [48]

    Robust change captioning

    Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 4624–4633, 2019. 3

    2019

  41. [49]

    It’s just another day: Unique video captioning by discriminative prompting

    Toby Perrett, Tengda Han, Dima Damen, and Andrew Zis- serman. It’s just another day: Unique video captioning by discriminative prompting. arXiv preprint arXiv:2410.11702,

    arXiv

  42. [50]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  43. [51]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  44. [52]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. 3, 5

    2024

  45. [53]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  46. [54]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 14313–14323, 2024. 3

    2024

  47. [55]

    Silhouettes: a graphical aid to the inter- pretation and validation of cluster analysis

    Peter J Rousseeuw. Silhouettes: a graphical aid to the inter- pretation and validation of cluster analysis. Journal of com- putational and applied mathematics, 20:53–65, 1987. 2

    1987

  48. [56]

    The change you want to see (now in 3d)

    Ragav Sachdeva and Andrew Zisserman. The change you want to see (now in 3d). In IEEE/CVF International Confer- ence on Computer Vision (ICCV) , pages 2060–2069, 2023. 3

    2023

  49. [57]

    Only time can tell: Discovering temporal data for temporal modeling

    Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In IEEE Winter Conference on Applications of Computer Vi- sion (WACV), pages 535–544, 2021. 8

    2021

  50. [58]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  51. [59]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1207–1216, 2019. 4, 5, 1, 2

    2019

  52. [60]

    Show and tell: Lessons learned from the 2015 mscoco image captioning challenge

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(4):652–663,

    2015

  53. [61]

    Tarsier: Recipes for training and evaluating large video description models

    Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024. 2, 3, 5

    2024 arXiv

  54. [62]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  55. [63]

    Vatex: A large-scale, high- quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high- quality multilingual dataset for video-and-language research. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 4581–4591, 2019. 2, 3

    2019

  56. [64]

    Videoagent: Long-form video understand- ing with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent. arXiv preprint arXiv:2403.10517, 2024. 8

    2024 arXiv

  57. [65]

    Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models

    Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406.16338, 2024. 3

    2024 arXiv

  58. [66]

    Unified coarse-to-fine alignment for video-text retrieval

    Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 2816–2827, 2023. 1

    2023

  59. [67]

    Cap4video: What can auxiliary captions do for text-video retrieval? In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10704– 10713, 2023

    Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10704– 10713, 2023

    2023

  60. [68]

    Cap4video++: Enhancing video understanding with auxiliary captions

    Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Cap4video++: Enhancing video understanding with auxiliary captions. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024. 1

    2024

  61. [69]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9777–9786,

  62. [70]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288–5296, 2016. 2, 3

    2016

  63. [71]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 5

    2024 arXiv

  64. [72]

    Learn- ing object state changes in videos: An open-world perspec- tive

    Zihui Xue, Kumar Ashutosh, and Kristen Grauman. Learn- ing object state changes in videos: An open-world perspec- tive. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 18493–18503, 2024. 3, 5, 8, 1, 2

    2024

  65. [73]

    Just ask: Learning to answer questions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 1686–1697,

  66. [74]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2023

  67. [75]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2, 3

    2023 arXiv

  68. [76]

    Image captioning with semantic attention

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4651–4659, 2016. 2

    2016

  69. [77]

    Fine-grained visual compar- isons with local learning

    Aron Yu and Kristen Grauman. Fine-grained visual compar- isons with local learning. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 192– 199, 2014. 3

    2014

  70. [78]

    Video paragraph captioning using hierarchical recurrent neural networks

    Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4584–4593,

  71. [79]

    Lever- aging video descriptions to learn video question answering

    Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Lever- aging video descriptions to learn video question answering. In Association for the Advancement of Artificial Intelligence (AAAI), 2017. 1

    2017

  72. [80]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023. 7

    2023

  73. [81]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2, 3

    2023 arXiv

  74. [82]

    From actemes to action: A strongly-supervised repre- sentation for detailed action understanding

    Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpa- nis. From actemes to action: A strongly-supervised repre- sentation for detailed action understanding. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 2248–2255, 2013. 5, 8, 1, 2

    2013

  75. [83]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 3, 6, 1

    2024 arXiv

  76. [84]

    Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. arXiv preprint arXiv:2311.16839, 2023. 3

    2023 arXiv

  77. [85]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Association for the Advancement of Artificial In- telligence (AAAI), 2018. 1

    2018

  78. [86]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Association for the Advancement of Artificial In- telligence (AAAI), 2018. 2, 3

    2018

  79. [87]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18243–18252, 2024. 3 Progress-Aware Video Frame ...

    2024

  80. [88]

    accepted

    Dataset 1.1. FrameCap Training Data To construct the FramePair dataset, we employ a suite of open-source VLMs as captioners for initial pseudo la- bel generation, including VILA [40], Qwen2-VL [62], LLA V A-Next-Video [35], LLA V A-Video [83] and LLA V A- OV [33]. Training vid...

  81. [89]

    changes” can en- compass broader aspects than “progression

    Experiments 2.1. Experimental setup Evaluation Metric Design Progression detection evalu- ates a model’s action progress awareness, using caption pairs generated for each frame pair. It functions as a bi- nary classification task, where label = 0 identifies scenarios with no v...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.