Pith. sign in

REVIEW 3 major objections 6 minor 108 references

Towards Long Video Understanding via Fine-detailed Video Story Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a long, untrimmed video can be compressed into hierarchical textual representations—clip chapters plus an overall story—that directly support zero-shot video retrieval, partially relevant retrieval, and question…

desk verdict A clever training-free video-to-text pipeline with broad experiments, but the reported numbers are undercut by test-set tuning, no error bars, and an implausible storage figure. read the letter →

arxiv 2412.06182 v2 pith:3M6CH4CJ submitted 2024-12-09 cs.CV

Zeng You , Zhiquan Wen , Yaofo Chen , Xin Li , Runhao Zeng , Yaowei Wang , Mingkui Tan This is my paper · ORCID
classification cs.CV
keywords longvideounderstandinghierarchicaltextualrepresentationzero-shotstorygenerationredundancyreductiontext-to-videoretrievalquestionansweringlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FDVS is a method for converting a long, untrimmed video into a compact hierarchy of text: a short chapter for each clip and one story for the whole video. The paper's central claim is that this textual representation is rich enough to feed directly into three downstream video tasks—text-to-video retrieval, partially relevant video retrieval, and video question answering—without any task-specific fine-tuning or training on video-text pairs. The method's central move is a bottom-up process of perception, clip-level summarization, and video-level summarization that keeps the details that matter while discarding visual and semantic redundancy. A reader should care because most long-video systems either need labelled data for each task or align vision and language with expensive pre-training, whereas FDVS reuses frozen components and leaves the result as human-readable text. The paper reports results on eight datasets spanning the three tasks, claiming that the zero-shot text representation matches or exceeds several models that were trained on video-text data.

What carries the argument

The load-bearing object is the Bottom-up Video Interpretation Mechanism, a hierarchical summarization chain in which each level's text is produced by a large language model from the previous level's text, working upward from frame-level perception to clip chapters to the whole-video story. Two redundancy operators support it: visual redundancy reduction removes frames whose image-encoder cosine similarity to the clip's keyframe is high, and textual redundancy reduction removes a chapter whose text-encoder similarity to the running average of recent chapters is high. The remaining chapters are then summarized into a single story, and downstream tasks consume either the story, selected chapters, or both.

What would settle it

Take a video with a known scene change that falls between two keyframes—for example, a long single take with no keyframe at the moment the location switches—and check whether FDVS's chapter for that clip describes both scenes; if it does, the segmentation premise fails.

Watch

Extended reading notes

Core claim

The central claim is that an untrimmed long video can be represented as a hierarchy of text with multi-granularity detail, and that this text alone is sufficient input for several video-understanding tasks. FDVS builds the hierarchy bottom-up: it segments the video at keyframes, prunes visually redundant frames within each clip, extracts object, action, and scene information with frozen perception models, has a language model write a chapter for each clip, prunes textually redundant chapters, and then has the language model summarize the surviving chapters into one long video story. The paper asserts that this hierarchical textual representation carries enough fine-grained and global information that retrieval, partially relevant retrieval, and question answering can be run on it in a zero-shot manner, with no fine-tuning on task-specific datasets.

Load-bearing premise

The load-bearing assumption is that the video encoder's keyframes land on real scene or event boundaries, so that each clip between two keyframes contains one coherent scene.

Editorial extensions

If this is right

  • A single offline pass over an untrimmed video yields a reusable text artifact that can answer retrieval, partially relevant retrieval, and question answering queries without any per-task training.
  • Video retrieval reduces to text-to-text retrieval, so any text embedding model can score queries against the story and the chapters.
  • Long-video question answering becomes a reading task for a language model over the story text, decoupling visual perception from language reasoning.
  • The paper reports average storage per video of 87.3 bytes on MSRVTT, far smaller than stored deep-feature representations, which would make large video archives cheaper to index.
  • Because all components are frozen, improving the perception models or the language model should directly improve downstream accuracy, with no extra fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implicit consequence not tested in the paper is that the same recipe could extend to audio: adding a speech recognizer as another perception agent would let the story text cover dialogue and sound events, a direction the paper itself mentions as a possible optimization.
  • Because the representation is plain text, the story and chapters could be indexed by standard text search engines, enabling retrieval over very large video collections; the paper demonstrates semantic text retrieval but not this systems-level deployment.
  • A testable extension is to feed the generated chapters to a temporal localization or summarization task by changing only the prompt template; if the representation is as task-agnostic as claimed, no other change should be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes FDVS, a pipeline that converts long untrimmed videos into hierarchical textual representations using only off-the-shelf pretrained modules: keyframe-based clip segmentation, sampled-frame visual redundancy reduction, three perception agents (object detection, action recognition, image captioning), an LLM that writes per-clip "chapters," a textual redundancy reduction stage, and a final LLM-generated "story." These text representations are then used directly, without task-specific fine-tuning, for partially relevant video retrieval, text-to-video retrieval, and video question answering. The method is evaluated on eight datasets across the three tasks, with ablations over sampling strategy, redundancy reduction, perception modules, local memory length, and number of sampled frames. The paper reports strong zero-shot numbers, lower storage costs than deep-feature baselines, and qualitative examples of detailed video understanding.

Significance. If the empirical claims hold, FDVS offers a practical and interpretable zero-shot alternative to learned video features: it introduces no trainable parameters, reuses strong vision and language foundation models, and provides a single hierarchical text artifact that supports retrieval, partially relevant retrieval, and QA. The paper also ships an unusually large set of ablations and reports storage advantages, which are useful for practitioners. However, the central "no fine-tuning" and "zero-shot" claims are currently weakened by test-set-based hyperparameter selection, single-run stochastic LLM decoding, and an unsupported assumption that encoder I-frames provide semantically coherent clip boundaries. These issues are fixable within the manuscript's scope, but they are load-bearing for the headline generalization claim.

major comments (3)
  1. [Section III-B and Section IV-F, Table XI] The keyframe-based segmentation is the foundation of the entire hierarchy, but the paper's justification conflates compression independence with semantic coherence. Section III-B argues that intra frames are reliable because they are "independent of inter-frame information," yet I-frame placement is determined by the encoder's GOP structure for random access, not by scene or event boundaries. A scene change can therefore fall inside one clip, and a long scene can be split across multiple clips. Table XI does not isolate this effect: the "All frames" and "Keyframe-based sampling" rows share the same keyframe-derived boundaries, while "Uniform sampling" changes both boundary placement and sampling density. I ask the authors to quantify clip coherence directly (e.g., compare with a scene detector such as PySceneDetect or with annotated boundaries) and to report how retrieval and QA scores change when boundaries are perturbed.
  2. [Section IV-B, Tables XVIII and XIX] The headline zero-shot claim is undermined by selecting the two main hyperparameters on the very test sets used to report the headline numbers. Section IV-B states that the local memory length L=35 is chosen "based on the ablation study in Table XVIII," which evaluates video retrieval on ActivityNet Captions, and the default 8 frames per clip is chosen from Table XIX on MSRVTT. These are the same datasets whose final retrieval results appear in Tables VIII and IX. Because the pipeline is otherwise parameter-free, tuning these choices on the test sets can absorb test-specific variance and overstate the zero-shot generalization. Please move hyperparameter selection to a validation split or use a leave-one-dataset-out protocol, and state the selected values explicitly.
  3. [Section IV-B and Tables IV-IX] All quantitative results appear to be single runs even though the LLM is decoded with temperature 0.7. Several headline comparisons are close enough to be within decoding noise (e.g., Table VIII: FDVS 31.6 R@1 vs. HiTeA 29.9; Table VI: FDVS 42.4 vs. Bard+ShortViViT 42.0; Table XVIII: L=31 vs. L=35 differ by 1.1 R@1). Without multiple seeds and variance estimates, or a switch to greedy decoding for the main tables, the reported improvements are not statistically grounded. Please report mean and standard deviation over at least three runs for the central tables, or use temperature 0 and state that the results are deterministic.
minor comments (6)
  1. [Section IV-F, Table XV] The caption of Table XV says the ablation is on ActivityNet Captions, but the default rows (CLIP-base, Vicuna-7B: R@1 31.6, R@5 56.8, R@10 65.3) match the MSRVTT numbers in Table VIII rather than the ActivityNet numbers in Table IX (R@1 28.4, R@5 57.5, R@10 71.1). Please correct either the caption or the numbers.
  2. [Table II] The prompt templates contain several typos that should be fixed: "finnally," "winthin," "infomation," and "explaination." Minor as they are, they matter because the prompts are part of the method's reproducibility.
  3. [Section IV-B and Table X] Table X reports storage in "bytes" per video (e.g., 87.3 bytes for FDVS on MSRVTT), but a detailed textual story is far larger than 87 bytes. Please clarify the unit (bytes, kilobytes, tokens, or normalized feature bytes) and state how the number was computed; otherwise the 89.6% storage reduction claim is not verifiable.
  4. [Section III-C, Eq. (2)] Eq. (2) defines a cosine similarity, but the text calls it a cosine distance and then removes frames whose "similarity distance" exceeds the threshold. This is confusing: a frame more similar to the keyframe is removed as redundant, which is consistent with the similarity interpretation, but the terminology should be made consistent throughout.
  5. [Section III-D, Eq. (4)] The symbol L is reused for both the number of chapters in Eq. (4) and the local memory length in Section III-E. This creates ambiguity when reading the ablation in Table XVIII. Please use distinct symbols (e.g., K for the number of chapters and L for memory length).
  6. [Tables IV-VIII] The column "#V-T Data" labels FDVS as 0M, which is technically true only for the FDVS pipeline itself. The pipeline uses InternVideo, BLIP2, and GroundingDINO, all of which were pretrained on large video-text or image-text corpora, and retrieval uses AnglE, which is fine-tuned on semantic textual similarity. The 0M label should be qualified so readers do not interpret the comparison as showing that no external data were used anywhere in the system.

Circularity Check

2 steps flagged · score 4.0 of 10

Headline zero-shot retrieval numbers use hyperparameters selected on the same benchmarks; the underlying pipeline is otherwise externally grounded and not circular.

  1. fitted input called prediction [Section IV-B (Experimental Setup) and Table XVIII, with final retrieval results in Table IX]
    "We set the length L of local memory used in semantic redundancy reduction as 35 based on the ablation study in Table XVIII."

    The paper selects L by maximizing video-retrieval SumR on ActivityNet Captions (Table XVIII) and then reports ActivityNet Captions zero-shot retrieval in Table IX using that same L. The ActivityNet R@1/R@5/R@10 numbers are therefore produced by a configuration tuned on the very benchmark they are used to validate, so the 'zero-shot prediction' on ActivityNet is partly an evaluation-set fit rather than an independent generalization result. This is not a forced identity, but it weakens the claim that the headline long-video retrieval numbers are parameter-free.

  2. fitted input called prediction [Section IV-F (Ablation study on the number of frames sampled per clip, Table XIX) and final MSRVTT retrieval in Table VIII]
    "The experiments indicate that a smaller sampling number (e.g., 4) results in performance degradation while increasing the sampling number to 8 yields improvements. However, further increases to 16 or 32 frames show only marginal performance gains. Thus, to balance computational load and performance, we have adopted a default sampling number of 8."

    The frames-per-clip count is chosen from the MSRVTT zero-shot retrieval ablation (Table XIX) and then the same MSRVTT zero-shot retrieval table (Table VIII) reports FDVS with that choice. The reported MSRVTT R@1=31.6 is thus not a prediction made before seeing the benchmark; it is the result of a configuration selected on that benchmark. The paper partly mitigates this by noting 8 frames is consistent with prior work, but the selection is still benchmark-informed.

full rationale

The core FDVS derivation is not circular: the video-to-story pipeline contains no learned parameters, and its perception modules (GroundingDINO, InternVideo, BLIP2), text encoders (CLIP, Sentence-BERT, AnglE), and LLM (Vicuna) are all externally pretrained components whose outputs are compared against external benchmarks. The hierarchical textual representation is generated by composition of these components and is not defined in terms of the downstream metrics, so the central zero-shot results do not reduce to the inputs by construction. The main circularity-adjacent issue is test-set hyperparameter selection: L=35 is chosen from the ActivityNet Captions retrieval ablation and frames-per-clip=8 from the MSRVTT retrieval ablation, and both choices are then used in the headline retrieval tables on those same datasets. This is a mild form of fitted-input-called-prediction for those two benchmarks, but it does not infect the QA or PRVR results, and no load-bearing self-citation chain is present. The I-frame segmentation concern is a correctness/robustness risk, not a circularity, and is therefore not scored as a circular step.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced; the method adds only a representation (hierarchical text). The free parameters are pipeline hyperparameters, two of which are tuned on the test benchmarks used for evaluation.

free parameters (3)
  • Local memory length L = 35
    Selected by ablation on ActivityNet Captions test set (Table XVIII), then used in all experiments. Tuning on test data inflates reported results.
  • Frames sampled per clip = 8
    Selected by ablation on MSRVTT test set (Table XIX), then used across all datasets. Higher values gave marginal gains.
  • GroundingDINO detection thresholds = box 0.4, text 0.25
    Chosen by the authors in Section IV-B; these thresholds affect the object-level information fed to the LLM and are not justified by validation.
assumptions (5)
  • domain assumption Intra frames in compressed video are reliable cues for scene or event boundaries.
    Section III-B argues I-frames avoid motion-vector errors, but GOP structure is encoder-dependent and may not align with semantic boundaries.
  • domain assumption CLIP cosine similarity between a frame and the key frame measures visual redundancy.
    Section III-C, Eq. (2); no validation that high similarity implies redundant content.
  • domain assumption Sentence-BERT similarity captures semantic redundancy between clip chapters.
    Section III-E; used to drop chapters, but no analysis of what similar means for narrative continuity.
  • domain assumption The three perception models provide sufficient information for the LLM to infer correct video content.
    Section III-D; the whole story generation rests on this, and errors in detection, action recognition, or captioning would propagate.
  • domain assumption AnglE embedding similarity between query and generated text is a valid retrieval score.
    Section IV-B; used for text-to-text retrieval, and assumes the text embedding is well aligned with the query and story semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Long Video Understanding via Fine-detailed Video Story Generation." pith.science (2026). https://pith.science/paper/3M6CH4CJ

@misc{pith2026241206182,
  author       = {Pith},
  title        = {Pith review of: Towards Long Video Understanding via Fine-detailed Video Story Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3M6CH4CJ}},
  note         = {Machine review of arXiv:2412.06182}
}
read the original abstract

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.

Figures

Figures reproduced from arXiv: 2412.06182 by the authors.

Figure 1
Figure 1. Illustration of video understanding with LLMs. To [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. General pipeline of our method. We extract a compact hierarchical textual representation rather than deep features for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of information extraction via three-level [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Qualitative results of our method and VideoLLaVa [93]. The videos are from ActivityNet Captions. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The category labels are obtained from the second [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 6
Figure 6. Figure 6: R@1 performance of FDVS across various video [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

108 extracted references · 62 canonical work pages

  1. [1]

    Intermediary-guided bidi- rectional spatial-temporal aggregation network for video-based visible- infrared person re-identification,

    H. Li, M. Liu, Z. Hu, F. Nie, and Z. Yu, “Intermediary-guided bidi- rectional spatial-temporal aggregation network for video-based visible- infrared person re-identification,” IEEE Transactions on Circuits and Systems for Video Technology, 2023

    2023

  2. [2]

    Video moment re- trieval via comprehensive relation-aware network,

    X. Sun, J. Gao, Y . Zhu, X. Wang, and X. Zhou, “Video moment re- trieval via comprehensive relation-aware network,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  3. [3]

    Self-supervised adversarial video summarizer with context latent sequence learning,

    Y . Xu, X. Li, L. Pan, W. Sang, P. Wei, and L. Zhu, “Self-supervised adversarial video summarizer with context latent sequence learning,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  4. [4]

    Complementarity- aware space learning for video-text retrieval,

    J. Zhu, P. Zeng, L. Gao, G. Li, D. Liao, and J. Song, “Complementarity- aware space learning for video-text retrieval,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  5. [5]

    Moma-lrg: Language-refined graphs for multi-object multi-actor activity parsing,

    Z. Luo, Z. Durante, L. Li, W. Xie, R. Liu, E. Jin, Z. Huang, L. Y . Li, J. Wu, J. C. Niebles et al. , “Moma-lrg: Language-refined graphs for multi-object multi-actor activity parsing,” Advances in Neural Information Processing Systems , vol. 35, pp. 5282–5298, 2022

    2022

  6. [6]

    Videoclip: Contrastive pre- training for zero-shot video-text understanding,

    H. Xu, G. Ghosh, P.-Y . Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer, “Videoclip: Contrastive pre- training for zero-shot video-text understanding,” in Proceedings of Conference on Empirical Methods in Natural Language Processing , 2021, pp. 6787–6800

    2021

  7. [7]

    Graph convolutional module for temporal action localization in videos,

    R. Zeng, W. Huang, M. Tan, Y . Rong, P. Zhao, J. Huang, and C. Gan, “Graph convolutional module for temporal action localization in videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6209–6223, 2022

    2022

  8. [8]

    Evcap: Element- aware video captioning,

    S. Liu, A. Li, Y . Zhao, J. Wang, and Y . Wang, “Evcap: Element- aware video captioning,” IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

Show all 108 references
  1. [9]

    Multi-granularity interaction and integration network for video question answering,

    Y . Wang, M. Liu, J. Wu, and L. Nie, “Multi-granularity interaction and integration network for video question answering,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 33, no. 12, pp. 7684–7695, 2023

    2023

  2. [10]

    Video question answering with semantic disentanglement and reasoning,

    J. Liu, G. Wang, J. Xie, F. Zhou, and H. Xu, “Video question answering with semantic disentanglement and reasoning,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  3. [11]

    Multilevel semantic interaction alignment for video–text cross-modal retrieval,

    L. Chen, Z. Deng, L. Liu, and S. Yin, “Multilevel semantic interaction alignment for video–text cross-modal retrieval,” IEEE Transactions on Circuits and Systems for Video Technology , 2024. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 14

    2024

  4. [12]

    Videochat: Chat-centric video understanding,

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video understanding,”arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  5. [13]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” in Proceedings of Conference on Empirical Methods in Natural Language Processing , 2023, pp. 543–553

    2023

  6. [14]

    Moviechat: From dense token to sparse memory for long video understanding,

    E. Song, W. Chai, G. Wang, Y . Zhang, H. Zhou, F. Wu, X. Guo, T. Ye, Y . Lu, J.-N. Hwang et al. , “Moviechat: From dense token to sparse memory for long video understanding,” arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  7. [15]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424 , 2023

    2023 arXiv

  8. [16]

    Language models with image descriptors are strong few-shot video-language learners,

    Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, S. Chang, M. Bansal, and H. Ji, “Language models with image descriptors are strong few-shot video-language learners,” in Advances in Neural Information Processing Systems , 2022

    2022

  9. [17]

    Compressed video action recognition with dual-stream and dual-modal transformer,

    Y . Mou, X. Jiang, K. Xu, T. Sun, and Z. Wang, “Compressed video action recognition with dual-stream and dual-modal transformer,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  10. [18]

    Dynamic spatial focus for efficient compressed video action recognition,

    Z. Zheng, L. Yang, Y . Wang, M. Zhang, L. He, G. Huang, and F. Li, “Dynamic spatial focus for efficient compressed video action recognition,” IEEE Transactions on Circuits and Systems for Video Technology, 2023

    2023

  11. [19]

    Alignment-guided temporal atten- tion for video action recognition,

    Y . Zhao, Z. Li, X. Guo, and Y . Lu, “Alignment-guided temporal atten- tion for video action recognition,” in Advances in Neural Information Processing Systems, 2022

    2022

  12. [20]

    Slowfast networks for video recognition,

    C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in IEEE International Conference on Computer Vision, 2019, pp. 6201–6210

    2019

  13. [21]

    Temporal distinct representation learning for action recognition,

    J. Weng, D. Luo, Y . Wang, Y . Tai, C. Wang, J. Li, F. Huang, X. Jiang, and J. Yuan, “Temporal distinct representation learning for action recognition,” in European Conference on Computer Vision . Springer, 2020, pp. 363–378

    2020

  14. [22]

    Truncate-split-contrast: a framework for learning from mislabeled videos,

    Z. Wang, J. Weng, C. Yuan, and J. Wang, “Truncate-split-contrast: a framework for learning from mislabeled videos,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 3, 2023, pp. 2751–2758

    2023

  15. [23]

    Reading-strategy inspired visual representation learning for text-to- video retrieval,

    J. Dong, Y . Wang, X. Chen, X. Qu, X. Li, Y . He, and X. Wang, “Reading-strategy inspired visual representation learning for text-to- video retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 8, pp. 5680–5694, 2022

    2022

  16. [24]

    Use what you have: Video retrieval using representations from collaborative experts,

    Y . Liu, S. Albanie, A. Nagrani, and A. Zisserman, “Use what you have: Video retrieval using representations from collaborative experts,” in British Machine Vision Conference , 2019, p. 279

    2019

  17. [25]

    Dual encoding for zero-example video retrieval,

    J. Dong, X. Li, C. Xu, S. Ji, Y . He, G. Yang, and X. Wang, “Dual encoding for zero-example video retrieval,” in IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 9346–9355

    2019

  18. [26]

    Locvtp: Video-text pre-training for temporal localization,

    M. Cao, T. Yang, J. Weng, C. Zhang, J. Wang, and Y . Zou, “Locvtp: Video-text pre-training for temporal localization,” in European Confer- ence on Computer Vision . Springer, 2022, pp. 38–56

    2022

  19. [27]

    Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization,

    R. Zeng, C. Gan, P. Chen, W. Huang, Q. Wu, and M. Tan, “Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization,” IEEE Transactions on Image Processing, vol. 28, no. 12, pp. 5797–5808, 2019

    2019

  20. [28]

    Cross time-frequency transformer for temporal action localization,

    J. Yang, P. Wei, and N. Zheng, “Cross time-frequency transformer for temporal action localization,” IEEE Transactions on Circuits and Systems for Video Technology, 2023

    2023

  21. [29]

    Slow motion matters: A slow motion enhanced network for weakly supervised temporal action localization,

    W. Sun, R. Su, Q. Yu, and D. Xu, “Slow motion matters: A slow motion enhanced network for weakly supervised temporal action localization,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 33, no. 1, pp. 354–366, 2022

    2022

  22. [30]

    Long-form video- language pre-training with multimodal temporal contrastive learning,

    Y . Sun, H. Xue, R. Song, B. Liu, H. Yang, and J. Fu, “Long-form video- language pre-training with multimodal temporal contrastive learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 38 032–38 045, 2022

    2022

  23. [31]

    Videograph: Rec- ognizing minutes-long human activities in videos,

    N. Hussein, E. Gavves, and A. W. Smeulders, “Videograph: Rec- ognizing minutes-long human activities in videos,” arXiv preprint arXiv:1905.05143, 2019

    1905 arXiv

  24. [32]

    Supervoxel attention graphs for long-range video modeling,

    Y . Wang, G. Bertasius, T.-H. Oh, A. Gupta, M. Hoai, and L. Torre- sani, “Supervoxel attention graphs for long-range video modeling,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 155–166

    2021

  25. [33]

    Long movie clip classification with state-space video models,

    M. M. Islam and G. Bertasius, “Long movie clip classification with state-space video models,” in European Conference on Computer Vision. Springer, 2022, pp. 87–104

    2022

  26. [34]

    S4nd: Modeling images and videos as multidimensional signals with state spaces,

    E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. R ´e, “S4nd: Modeling images and videos as multidimensional signals with state spaces,” Advances in Neural Information Processing Systems, vol. 35, pp. 2846–2861, 2022

    2022

  27. [35]

    Selective structured state-spaces for long-form video understanding,

    J. Wang, W. Zhu, P. Wang, X. Yu, L. Liu, M. Omar, and R. Hamid, “Selective structured state-spaces for long-form video understanding,” in IEEE Conference on Computer Vision and Pattern Recognition , 2023, pp. 6387–6397

    2023

  28. [36]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. Re, “Efficiently modeling long sequences with structured state spaces,” in International Conference on Learning Representations, 2021

    2021

  29. [37]

    Mgsampler: An explainable sampling strategy for video action recognition,

    Y . Zhi, Z. Tong, L. Wang, and G. Wu, “Mgsampler: An explainable sampling strategy for video action recognition,” in IEEE International Conference on Computer Vision , 2021, pp. 1513–1522

    2021

  30. [38]

    Adaframe: Adaptive frame selection for fast video recognition,

    Z. Wu, C. Xiong, C.-Y . Ma, R. Socher, and L. S. Davis, “Adaframe: Adaptive frame selection for fast video recognition,” in IEEE Confer- ence on Computer Vision and Pattern Recognition , 2019, pp. 1278– 1287

    2019

  31. [39]

    Localizing moments in long video via multimodal guidance,

    W. Barrios, M. Soldan, A. M. Ceballos-Arroyo, F. C. Heilbron, and B. Ghanem, “Localizing moments in long video via multimodal guidance,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 13 667–13 678

    2023

  32. [40]

    Mad: A scalable dataset for language grounding in videos from movie audio descriptions,

    M. Soldan, A. Pardo, J. L. Alc ´azar, F. Caba, C. Zhao, S. Giancola, and B. Ghanem, “Mad: A scalable dataset for language grounding in videos from movie audio descriptions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5026–5035

    2022

  33. [41]

    End-to-end learning of visual representations from uncurated instructional videos,

    A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zis- serman, “End-to-end learning of visual representations from uncurated instructional videos,” in IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 9879–9889

    2020

  34. [42]

    Merlot: Multimodal neural script knowledge models,

    R. Zellers, X. Lu, J. Hessel, Y . Yu, J. S. Park, J. Cao, A. Farhadi, and Y . Choi, “Merlot: Multimodal neural script knowledge models,” Advances in Neural Information Processing Systems , vol. 34, pp. 23 634–23 651, 2021

    2021

  35. [43]

    Scaling up vision-language pre-training for image captioning,

    X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y . Lu, and L. Wang, “Scaling up vision-language pre-training for image captioning,” in IEEE Conference on Computer Vision and Pattern Recognition , 2022, pp. 17 980–17 989

    2022

  36. [44]

    Gbc: Guided alignment and adaptive boosting clip bridging vision and language for robust action recognition,

    Z. Yang, G. An, Z. Zheng, S. Cao, and Q. Ruan, “Gbc: Guided alignment and adaptive boosting clip bridging vision and language for robust action recognition,” IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

  37. [45]

    Unsu- pervised pre-training for temporal action localization tasks,

    C. Zhang, T. Yang, J. Weng, M. Cao, J. Wang, and Y . Zou, “Unsu- pervised pre-training for temporal action localization tasks,” in IEEE Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 031–14 041

    2022

  38. [46]

    Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning,

    H. Luo, L. Ji, M. Zhong, Y . Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning,” Neurocomputing, vol. 508, pp. 293–304, 2022

    2022

  39. [47]

    Human action recognition and prediction: A survey,

    Y . Kong and Y . Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision , vol. 130, no. 5, pp. 1366–1401, 2022

    2022

  40. [48]

    A survey on video-based human action recognition: recent updates, datasets, challenges, and applications,

    P. Pareek and A. Thakkar, “A survey on video-based human action recognition: recent updates, datasets, challenges, and applications,” Artificial Intelligence Review , vol. 54, pp. 2259–2322, 2021

    2021

  41. [49]

    Lavender: Unifying video-language understanding as masked lan- guage modeling,

    L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and L. Wang, “Lavender: Unifying video-language understanding as masked lan- guage modeling,” in IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 23 119–23 129

    2023

  42. [50]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training,” Advances in Neural Information Processing Systems, vol. 35, pp. 10 078–10 093, 2022

    2022

  43. [51]

    Videomae v2: Scaling video masked autoencoders with dual masking,

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y . He, Y . Wang, Y . Wang, and Y . Qiao, “Videomae v2: Scaling video masked autoencoders with dual masking,” in IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 549–14 560

    2023

  44. [52]

    Violet: End-to-end video-language transformers with masked visual- token modeling,

    T.-J. Fu, L. Li, Z. Gan, K. Lin, W. Y . Wang, L. Wang, and Z. Liu, “Violet: End-to-end video-language transformers with masked visual- token modeling,” arXiv preprint arXiv:2111.12681 , 2021

    2021 arXiv

  45. [53]

    All in one: Exploring unified video-language pre-training,

    J. Wang, Y . Ge, R. Yan, Y . Ge, K. Q. Lin, S. Tsutsui, X. Lin, G. Cai, J. Wu, Y . Shan et al. , “All in one: Exploring unified video-language pre-training,” in IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 6598–6608. IEEE TRANSACTIONS ON CIRCUITS AND S...

    2023

  46. [54]

    Internvideo: General video foundation models via generative and discriminative learning,

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wang et al., “Internvideo: General video foundation models via generative and discriminative learning,” arXiv preprint arXiv:2212.03191, 2022

    2022 arXiv

  47. [55]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

    1901

  48. [56]

    GLM: general language model pretraining with autoregressive blank infilling,

    Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “GLM: general language model pretraining with autoregressive blank infilling,” pp. 320–335, 2022

    2022

  49. [57]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  50. [58]

    MIST : Multi-modal iterative spatial-temporal transformer for long-form video question answering,

    D. Gao, L. Zhou, L. Ji, L. Zhu, Y . Yang, and M. Z. Shou, “MIST : Multi-modal iterative spatial-temporal transformer for long-form video question answering,” in IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 773–14 783

    2023

  51. [59]

    A joint sequence fusion model for video question answering and retrieval,

    Y . Yu, J. Kim, and G. Kim, “A joint sequence fusion model for video question answering and retrieval,” in European Conference on Computer Vision, vol. 11211, 2018, pp. 487–503

    2018

  52. [60]

    Video question answering via gradually refined attention over appear- ance and motion,

    D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y . Zhuang, “Video question answering via gradually refined attention over appear- ance and motion,” in Proceedings of ACM international conference on Multimedia, 2017, pp. 1645–1653

    2017

  53. [61]

    Activitynet-qa: A dataset for understanding complex web videos via question answering,

    Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y . Zhuang, and D. Tao, “Activitynet-qa: A dataset for understanding complex web videos via question answering,” in AAAI Conference on Artificial Intelligence , 2019, pp. 9127–9134

    2019

  54. [62]

    Swinbert: End-to-end transformers with sparse attention for video captioning,

    K. Lin, L. Li, C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y . Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” in IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 928–17 937

    2022

  55. [63]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning,

    A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid, “Vid2seq: Large-scale pretraining of a visual language model for dense video captioning,” in IEEE Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 714–10 726

    2023

  56. [64]

    Dense- captioning events in videos,

    R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense- captioning events in videos,” in IEEE International Conference on Computer Vision, 2017, pp. 706–715

    2017

  57. [65]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International Conference on Machine Learning , vol. 202, 23–29 Jul 2023, pp. 19 730–19 742

    2023

  58. [66]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: En- hancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

    2023 arXiv

  59. [67]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation,

    Y . Wang, Y . He, Y . Li, K. Li, J. Yu, X. Ma, X. Chen, Y . Wang, P. Luo, Z. Liu et al., “Internvid: A large-scale video-text dataset for multimodal understanding and generation,” arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  60. [68]

    Pyscenedetect,

    “Pyscenedetect,” https://github.com/Breakthrough/PySceneDetect

  61. [69]

    Decord: An efficient video loader for deep learning,

    “Decord: An efficient video loader for deep learning,” https://github.com/dmlc/decord

  62. [70]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning , 2021, pp. 8748–8763

    2021

  63. [71]

    DINO: DETR with improved denoising anchor boxes for end-to-end object detection,

    H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H. Shum, “DINO: DETR with improved denoising anchor boxes for end-to-end object detection,” in International Conference on Learning Representations, 2023

    2023

  64. [72]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of Conference on Empir- ical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing , 2019, pp. 3980–3990

    2019

  65. [73]

    Partially relevant video retrieval,

    J. Dong, X. Chen, M. Zhang, X. Yang, S. Chen, X. Li, and X. Wang, “Partially relevant video retrieval,” in Proceedings of ACM Interna- tional Conference on Multimedia , 2022, pp. 246–257

    2022

  66. [74]

    Joint searching and grounding: Multi-granularity video content retrieval,

    Z. Chen, X. Jiang, X. Xu, Z. Cao, Y . Mo, and H. T. Shen, “Joint searching and grounding: Multi-granularity video content retrieval,” in Proceedings of ACM International Conference on Multimedia , 2023, pp. 975–983

    2023

  67. [75]

    Hit: Hierar- chical transformer with momentum contrast for video-text retrieval,

    S. Liu, H. Fan, S. Qian, Y . Chen, W. Ding, and Z. Wang, “Hit: Hierar- chical transformer with momentum contrast for video-text retrieval,” in IEEE International Conference on Computer Vision, 2021, pp. 11 895– 11 905

    2021

  68. [76]

    Multi-modal trans- former for video retrieval,

    V . Gabeur, C. Sun, K. Alahari, and C. Schmid, “Multi-modal trans- former for video retrieval,” in European Conference on Computer Vision, vol. 12349, 2020, pp. 214–229

    2020

  69. [77]

    Cross-modal and hierarchical modeling of video and text,

    B. Zhang, H. Hu, and F. Sha, “Cross-modal and hierarchical modeling of video and text,” in European Conference on Computer Vision , vol. 11217, 2018, pp. 385–401

    2018

  70. [78]

    Eclipse: Efficient long- range video retrieval using sight and sound,

    Y . Lin, J. Lei, M. Bansal, and G. Bertasius, “Eclipse: Efficient long- range video retrieval using sight and sound,” in European Conference on Computer Vision , vol. 13694, 2022, pp. 413–430

    2022

  71. [79]

    TALL: temporal activity localization via language query,

    J. Gao, C. Sun, Z. Yang, and R. Nevatia, “TALL: temporal activity localization via language query,” in IEEE International Conference on Computer Vision, 2017, pp. 5277–5285

    2017

  72. [80]

    Hollywood in homes: Crowdsourcing data collection for activity understanding,

    G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in European Conference on Computer Vision , 2016, pp. 510–526

    2016

  73. [81]

    MSR-VTT: A large video descrip- tion dataset for bridging video and language,

    J. Xu, T. Mei, T. Yao, and Y . Rui, “MSR-VTT: A large video descrip- tion dataset for bridging video and language,” in IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 5288–5296

    2016

  74. [82]

    Question generation via overgenerating transformations and ranking,

    M. Heilman and N. A. Smith, “Question generation via overgenerating transformations and ranking,” DTIC Document, 2009

    2009

  75. [83]

    Egoschema: A diagnostic benchmark for very long-form video language understanding,

    K. Mangalam, R. Akshulakov, and J. Malik, “Egoschema: A diagnostic benchmark for very long-form video language understanding,” in Ad- vances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., 2023

    2023

  76. [84]

    Ego4d: Around the world in 3, 000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V . Cartillier, S. Crane, T. D...

    2022

  77. [85]

    Next-qa: Next phase of question-answering to explaining temporal actions,

    J. Xiao, X. Shang, A. Yao, and T. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in IEEE Confer- ence on Computer Vision and Pattern Recognition , 2021, pp. 9777– 9786

    2021

  78. [86]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6299–6308

    2017

  79. [87]

    Judging llm-as-a-judge with mt- bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt- bench and chatbot arena,” arXiv preprint arXiv:2306.05685 , 2023

    2023 arXiv

  80. [88]

    Sharegpt: Share your wildest chatgpt conversations with one click,

    “Sharegpt: Share your wildest chatgpt conversations with one click,” 2023, https://github.com/domeccleston/sharegpt

    2023

  81. [89]

    Angle-optimized text embeddings,

    X. Li and J. Li, “Angle-optimized text embeddings,” arXiv preprint arXiv:2309.12871, 2023

    2023 arXiv

  82. [90]

    Video corpus moment retrieval with contrastive learning,

    H. Zhang, A. Sun, W. Jing, G. Nan, L. Zhen, J. T. Zhou, and R. S. M. Goh, “Video corpus moment retrieval with contrastive learning,” in Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval , 2021, pp. 685–695

    2021

  83. [91]

    TVR: A large-scale dataset for video-subtitle moment retrieval,

    J. Lei, L. Yu, T. L. Berg, and M. Bansal, “TVR: A large-scale dataset for video-subtitle moment retrieval,” in European Conference on Computer Vision , vol. 12366, 2020, pp. 447–463

    2020

  84. [92]

    Dual learning with dynamic knowledge distillation for partially relevant video retrieval,

    J. Dong, M. Zhang, Z. Zhang, X. Chen, D. Liu, X. Qu, X. Wang, and B. Liu, “Dual learning with dynamic knowledge distillation for partially relevant video retrieval,” in IEEE International Conference on Computer Vision , 2023

    2023

  85. [93]

    Video-llava: Learning united visual representation by alignment before projection,

    B. Lin, B. Zhu, Y . Ye, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122 , 2023

    2023 arXiv

  86. [94]

    Just ask: Learning to answer questions from millions of narrated videos,

    A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Just ask: Learning to answer questions from millions of narrated videos,” in IEEE International Conference on Computer Vision , 2021, pp. 1666– 1677

    2021

  87. [95]

    MERLOT RESERVE: neural script knowledge through vision and language and sound,

    R. Zellers, J. Lu, X. Lu, Y . Yu, Y . Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y . Choi, “MERLOT RESERVE: neural script knowledge through vision and language and sound,” in IEEE IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 16 Conference on Com...

    2022

  88. [96]

    Zero-shot video question answering via frozen bidirectional language models,

    A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” in Advances in Neural Information Processing Systems , 2022

    2022

  89. [97]

    Hitea: Hierarchical temporal-aware video-language pre-training,

    Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, and F. Huang, “Hitea: Hierarchical temporal-aware video-language pre-training,” in IEEE International Conference on Computer Vision, 2023, pp. 15 405– 15 416

    2023

  90. [98]

    Self-chained image-language model for video localization and question answering,

    S. Yu, J. Cho, P. Yadav, and M. Bansal, “Self-chained image-language model for video localization and question answering,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  91. [99]

    Memory consolidation enables long-context video understanding,

    I. Balazevic, Y . Shi, P. Papalampidi, R. Chaabouni, S. Koppula, and O. J. Henaff, “Memory consolidation enables long-context video understanding,” in International Conference on Machine Learning, vol. 235, 2024, pp. 2527–2542

    2024

  92. [100]

    mplug-owl: Modularization empowers large language models with multimodality,

    Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y . Zhou, J. Wang, A. Hu, P. Shi, Y . Shi et al. , “mplug-owl: Modularization empowers large language models with multimodality,” arXiv preprint arXiv:2304.14178 , 2023

    2023 arXiv

  93. [101]

    A simple LLM framework for long-range video question- answering,

    C. Zhang, T. Lu, M. M. Islam, Z. Wang, S. Yu, M. Bansal, and G. Bertasius, “A simple LLM framework for long-range video question- answering,” CoRR, vol. abs/2312.17235, 2023

    2023 arXiv

  94. [102]

    Revisiting the

    S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles, “Revisiting the” video” in video-language understanding,” in IEEE Conference on Computer Vision and Pattern Recognition , 2022, pp. 2917–2927

    2022

  95. [103]

    Video graph transformer for video question answering,

    J. Xiao, P. Zhou, T.-S. Chua, and S. Yan, “Video graph transformer for video question answering,” in European Conference on Computer Vision. Springer, 2022, pp. 39–58

    2022

  96. [104]

    Verbs in action: Improving verb understanding in video-language models,

    L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid, “Verbs in action: Improving verb understanding in video-language models,” in IEEE International Conference on Computer Vision , 2023, pp. 15 579–15 591

    2023

  97. [105]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022

    2022

  98. [106]

    Videoagent: Long- form video understanding with large language model as agent,

    X. Wang, Y . Zhang, O. Zohar, and S. Yeung-Levy, “Videoagent: Long- form video understanding with large language model as agent,” arXiv preprint arXiv:2403.10517, 2024

    2024 arXiv

  99. [107]

    Align and prompt: Video-and-language pre-training with entity prompts,

    D. Li, J. Li, H. Li, J. C. Niebles, and S. C. H. Hoi, “Align and prompt: Video-and-language pre-training with entity prompts,” in IEEE Conference on Computer Vision and Pattern Recognition , 2022, pp. 4943–4953

    2022

  100. [108]

    Revealing single frame bias for video-and-language learning,

    J. Lei, T. L. Berg, and M. Bansal, “Revealing single frame bias for video-and-language learning,” in Proceedings of Annual Meeting of the Association for Computational Linguistics , 2023, pp. 487–507

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.