Pith. sign in

REVIEW 3 major objections 5 minor 78 references

GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GEXIA claims multi-grained video-language alignment can be learned from a single-grained dataset by concatenation, summarization, and iterative approximation, yielding long-form video understanding from short-clip pretraining.

desk verdict GEX is a genuinely useful data-expansion idea and IAM is a reasonable module, but the long-form gains are confounded by input length and #iter is chosen after seeing test numbers. read the letter →

arxiv 2412.07704 v1 pith:6P2VRVAX submitted 2024-12-10 cs.CV

Yicheng Wang , Zhikang Zhang , Jue Wang , David Fan , Zhenlin Xu , Linda Liu , Xiang Hao , Vimal Bhat
show 1 more author
Xinyu Li
This is my paper · ORCID
classification cs.CV
keywords video-languagepretrainingmulti-grainedvideo-textalignmentgranularityexpansioniterativeapproximationlong-formvideounderstandingcontrastivelearningretrievaltextsummarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-text data in the wild is multi-grained: videos run from seconds to hours, and the text paired with a video can be a one-line caption or a full paragraph. Existing pretraining datasets are effectively single-grained, and most models align video and text at one fixed level of detail. This paper proposes GEXIA, a two-part recipe: Granularity Expansion (GEX) turns a single-grained dataset into a multi-grained one by concatenating same-source short clips and texts into long clips and by summarizing the long texts into short ones, and an Iterative Approximation Module (IAM) compresses variable-length video and text features into fixed-size embeddings in a shared low-dimensional space. Trained only on short video clips, the resulting model reports state-of-the-art or comparable results on seven benchmarks and does especially well on long-form video tasks, which the paper attributes to the multi-grained pretraining data and granularity-aware iteration.

What carries the argument

The load-bearing machinery is the pair of GEX and IAM. GEX (Granularity Expansion) takes an existing single-grained dataset and produces new granularities without new data collection: Video Integration and Text Integration concatenate videos and texts that share a source ID, in time order, forming long-video-long-text pairs, and Text Compression uses a large language model to summarize the long text into short text, forming long-video-short-text pairs. IAM (Iterative Approximation Module) is the model component that absorbs variable-length dense features: a learnable base embedding serves as the query in a cross-attention block whose key and value come from the dense feature, followed by a self-attention block, and the whole block is repeated a number of times (#iter) that is set according to the input granularity, with longer videos and texts receiving more iterations. The output is a fixed-size low-dimensional embedding per modality, so conventional contrastive learning can align any number of granularities in one space.

What would settle it

Train the same GEXIA model twice on the same amount of GEX-expanded data, once with concatenated clips kept in their original time order and once with the order shuffled within each source video, then compare zero-shot retrieval on ActivityNet Captions. If shuffled-order pretraining performs as well as time-ordered pretraining, the temporal-coherence assumption that carries the long-form claim is not doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that multi-grained video-language alignment can be learned from a single-grained dataset and that this is what unlocks long-form video understanding. It argues that GEX-generated long-video-long-text and long-video-short-text pairs make the model aware of granularity, while the IAM allows one model to embed any granularity into the same low-dimensional space by running more iteration steps on longer inputs. The authors report state-of-the-art or comparable performance on cross-modal retrieval (ActivityNet, MSR-VTT, LSMDC), video classification (LVU, COIN), egocentric action classification (Charades-Ego), and video question answering (How2QA), with the largest gains on long-video benchmarks, and they show that a model pretrained on only short clips can outperform methods pretrained on long-video datasets.

Load-bearing premise

The method's long-form gains rest on the assumption that concatenating short clips from the same source in time order produces synthetic long videos similar enough to naturally occurring long videos that pretraining on them transfers to real long-video tasks.

Editorial extensions

If this is right

  • If the central claim holds, long-form video understanding can be improved without constructing a large naturally long-video pretraining dataset; stitching short clips and summarizing texts suffices.
  • A single pretrained model can be adapted to a new granularity at inference time by changing #iter, as the paper demonstrates on image-text data with #iter = 0 for the video branch.
  • The GEX operations are recursive and have no limit on the number of granularities, so the pipeline can in principle produce arbitrarily layered video-text alignments from one dataset.
  • Because the video and text encoders stay separate until the final contrastive loss, the approach can reuse existing image and text encoders, keeping retrieval complexity at O(Nv Nt).
  • The gains on ActivityNet, LVU, and COIN suggest that granularity-aware data expansion can compensate for a much smaller pretraining corpus (10M pairs versus 136M in some baselines).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The ablations show random concatenation already improves over single-grained pretraining, which suggests part of the gain comes from simply feeding the model longer context, and only a smaller part from temporal coherence within the original video; a follow-up study that separates these two factors would clarify the mechanism.
  • Editorial inference: A natural stress test is to shuffle the order of concatenated clips during pretraining; if order does not matter, the model is learning a bag-of-clips representation rather than temporal sequence, which would narrow the claim about long-form understanding.
  • Editorial inference: The same expansion-and-approximate recipe could be applied to other paired modalities with variable granularity, such as audio-transcript pairs or image sets with captions, since neither GEX nor IAM depends on video-specific structure.
  • Editorial inference: The paper evaluates long-form gains on benchmark datasets whose videos were originally collected as continuous recordings; the synthetic concatenation assumption would be tested more sharply on genuinely continuous, causally structured long videos, where a stitched clip sequence lacks narrative dependencies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GEXIA, a video-language pretraining framework with two components: GEX, a data pipeline that expands a single-grained dataset into multiple granularities by temporal concatenation (Integration) and LLM summarization (Compression), and IAM, an iterative cross-attention module that maps variable-length dense features to fixed-size embeddings with an input-length-dependent iteration count. The model is pretrained on InternVid-10M-FLT and evaluated on retrieval (ActivityNet, MSR-VTT, LSMDC), classification (LVU, COIN, Charades-Ego), and VQA (How2QA). The paper reports state-of-the-art or comparable results and highlights long-form video understanding gains.

Significance. If the results are validated, the GEX pipeline is a low-cost way to create multi-grained pretraining data without data collection, and the IAM offers a straightforward architecture for cross-modal alignment across varying input lengths. The strong results on COIN and LVU suggest the approach has practical merit. However, the key claim of long-form superiority is currently confounded by uncontrolled input-length differences between GEXIA and the baselines, and the #iter hyperparameter appears to be selected on test metrics. These issues must be resolved before the headline claims can be accepted.

major comments (3)
  1. [Section 4.1, Table 1, Table 3] The central long-form claim is confounded by input length. GEXIA uses d=32 frames for long videos and the full CLIP text context (77 tokens), whereas the compared baselines (e.g., ViCLIP, CLIP4Clip) typically sample 8–12 frames and truncate text to 32 tokens. Under this confound, the 20-point zero-shot T2V gain on ActivityNet over ViCLIP (35.3 vs 15.1) cannot be attributed to the proposed GEX/IAM method. The paper's own ablation in Table 3 shows that, with the same input length (and #iter=1), adding the multi-grained LVLT/LVST data improves T2V R@1 by only 1.5 points (41.6 to 43.1). To support the 'excels in long-form' claim, the authors should compare against a plain CLIP/mean-pooling baseline that uses the same d=32 frames and 77-token text, and should include a no-IAM (or #iter=0) ablation on the same dense features.
  2. [Section 4.4, Table 4b] The per-dataset selection of #iter is made on the reported test metrics, and the stated rule is not consistently supported by the data. On LSMDC zero-shot T2V R@1, the 3-3 setup (14.1) outperforms the chosen 1-1 setup (12.9), which contradicts the claim that #iter should be higher for longer inputs and lower for short ones. Since the authors do not report results on a validation split, the reported numbers—especially the zero-shot and finetuned exceptions—are likely optimistic. The authors should either pre-register a fixed rule for #iter (e.g., based on input length thresholds) and report a validation-based selection, or provide error bars over multiple seeds.
  3. [Section 3.1, Table 6] The GEX assumption that temporally concatenating same-source clips yields realistic long videos is not validated. The ablation in Table 6 shows random concatenation also gives a large gain over SVST only (44.8 vs 41.6 T2V R@1), while the additional gain from source-ID ordering is small (45.3 vs 44.8). This suggests that simply presenting the model with longer visual input drives most of the improvement, and the proposed integration strategy adds little. The paper should analyze the synthetic long videos (e.g., narrative continuity, shot distribution) and, more importantly, run a controlled experiment that uses the SVST-only pretrained model with d=32 at inference on long-video benchmarks to determine whether GEX is actually necessary for the long-form gains.
minor comments (5)
  1. [Section 3.2] The phrase 'A#iter V angA#iter T' contains a typo; it should be 'A#iter V and A#iter T'.
  2. [Section 4.2] The number of finetuning epochs is inconsistent: 50 epochs for retrieval in Section 4.2, but 100 epochs for other downstream tasks in Section 4.3; please clarify.
  3. [Figure 3] The y-axis ranges are narrow (34.0–35.5 for R@1), which visually exaggerates the effect; include error bars or start the axis at zero.
  4. [Table 1] The grouping of rows into 'Zero-shot', 'Finetuned', 'Model with Similar FLOPs', and 'Retrieval-specific Model' is confusing because the categories overlap; consider a clearer structure or a separate table.
  5. [Reproducibility] The paper does not mention code or data release; a reproducibility statement would be helpful.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: GEXIA is an empirical framework with ordinary hyperparameter tuning; self-citations are not load-bearing.

full rationale

The paper's derivation chain is empirical rather than deductive. GEX constructs multi-grained pretraining data by concatenating same-source clips and LLM summarization (Section 3.1); IAM maps dense features to fixed-size embeddings via iterative cross-attention (Section 3.2); contrastive learning aligns the embeddings. The claimed long-form improvements are evaluated on real benchmarks (ActivityNet, LVU, COIN) that are external to the pretraining data construction, so the central claim is not defined in terms of the method's own outputs. The ablation in Table 3 fixes #iter=1 and still shows gains from adding LVLT and LVST, isolating the GEX data contribution. Tables 4a and 4b test different pretraining and inference #iter settings; the choice of #iter=3 for long inputs is an empirically selected hyperparameter, not a quantity derived from or identical to the reported result. Selecting #iter on ActivityNet and then reporting ActivityNet results is a tuning/benchmark-fitting concern rather than logical circularity, and the paper does disclose the selection procedure. The comparison against baselines that use fewer frames or shorter text is a possible confound for the magnitude of the long-form gains, but that is an experimental control issue, not a circular-reasoning structure. Self-citations (e.g., [10], [16], [28], [57], [60]) appear as related-work context or baseline references and do not carry the load of the main argument; no uniqueness theorem or ansatz is smuggled in via self-citation. Overall, no step reduces by construction, so circularity is minimal.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three main assumptions: the coherence of concatenated videos, the fidelity of LLM summarization, and the benefit of more iterations for longer inputs. These are empirical modeling assumptions rather than unfounded inventions. The #iter hyperparameter is a free parameter tuned to evaluation datasets, which affects the strength of the reported results.

free parameters (3)
  • Iteration count #iter (video, text) = 1-1 for short-video/short-text, 3-3 for long-video/long-text, 3-1 for long-video/short-text, 0-1 for image-text
    Chosen per input granularity based on average video/text lengths and validated by ablations on ActivityNet, MSRVTT, LSMDC, Charades-Ego. This is the main adaptive parameter of IAM.
  • Number of base vectors N and embedding dimension D = Not reported in the paper
    Architecture hyperparameters of IAM that are not listed, so the model is under-specified for replication.
  • Selection threshold for SVST pairs (>=4 pairs per source) = 4
    Used to select 4.5M of 10M pairs for integration; chosen without a stated criterion.
assumptions (3)
  • domain assumption Concatenating clips from the same source in time order produces semantically coherent long videos.
    Section 3.1 and Figure 1; this assumption underlies GEX and the claimed transfer to natural long-form videos.
  • domain assumption LLM summarization preserves the video-text alignment for the compressed text.
    Section 3.1 and Appendix B; they verify with CLIP cosine similarity on 100 samples, but this is a modeling assumption.
  • ad hoc to paper More iterations in IAM lead to better approximation and alignment for longer inputs.
    Section 3.2; this is introduced as the design principle of IAM and is validated by ablations, but it is not a proven theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning." pith.science (2026). https://pith.science/paper/6P2VRVAX

@misc{pith2026241207704,
  author       = {Pith},
  title        = {Pith review of: GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-grained Video-language Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6P2VRVAX}},
  note         = {Machine review of arXiv:2412.07704}
}
read the original abstract

In various video-language learning tasks, the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from two crucial perspectives: data and modeling. Given the absence of a multi-grained video-text pretraining dataset, we introduce a Granularity EXpansion (GEX) method with Integration and Compression operations to expand the granularity of a single-grained dataset. To better model multi-grained data, we introduce an Iterative Approximation Module (IAM), which embeds multi-grained videos and texts into a unified, low-dimensional semantic space while preserving essential information for cross-modal alignment. Furthermore, GEXIA is highly scalable with no restrictions on the number of video-text granularities for alignment. We evaluate our work on three categories of video tasks across seven benchmark datasets, showcasing state-of-the-art or comparable performance. Remarkably, our model excels in tasks involving long-form video understanding, even though the pretraining dataset only contains short video clips.

Figures

Figures reproduced from arXiv: 2412.07704 by the authors.

Figure 1
Figure 1. An overview of the Granularity EXpansion (GEX) pipeline, which expands a single-grained dataset into a multi-grained dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of GEXIA, which consists of Granularity EXpansion (GEX), Dense Feature Extraction, Iterative Approximation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Zero-shot T2V retrieval results with different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Average model inference run time on ActivityNet [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 4
Figure 4. Figure 4: t-SNE visualization of the CLIP-based features for the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: The visualization of alignment scores for long-video-long-text and long-video-short-text pairs. Given the same video, our GEXIA [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The visualization of alignment scores for different [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 56 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkow...

    arXiv 2022

  2. [2]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 6836–6846,

  3. [3]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23066–23078, 2023. 1, 2, 3, 7

    work page 2023

  4. [4]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 1728–1738,

  5. [5]

    Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021. 7

    work page 2021

  6. [6]

    Blinkdl/rwkv-lm

    PENG Bo. Blinkdl/rwkv-lm. https://doi.org/10. 5281/zenodo.5196577, Aug. 2021. 3

    work page 2021

  7. [7]

    Revisiting the” video” in video-language understanding

    Shyamal Buch, Crist ´obal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the” video” in video-language understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2917–2927, 2022. 7

    work page 2022

  8. [8]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 1

    2015

Show all 78 references
  1. [9]

    Locvtp: Video-text pre-training for temporal localization

    Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Locvtp: Video-text pre-training for temporal localization. In European Conference on Computer Vision, 2022. 6, 4

    2022

  2. [10]

    Metaxas, and Hongxia Yang

    Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N. Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learn- ing: From patch and token embeddings to finite discrete to- kens. In Proceedings of the IEEE/CVF Conference on...

    2023

  3. [11]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt- 4 with 90%* chatgpt quality. https://lmsys.org/ blog/2023-...

    2023

  4. [12]

    Unsupervised and semi-supervised domain adaptation for action recognition from drones

    Jinwoo Choi, Gaurav Sharma, Manmohan Chandraker, and Jia-Bin Huang. Unsupervised and semi-supervised domain adaptation for action recognition from drones. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1717–1726, 2020. 7

    2020

  5. [13]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm. https: / / www . databricks . com / blog / 2023 / 04 / 12 / dolly - ...

    2023

  6. [14]

    Prompt switch: Efficient clip adaptation for text-video re- trieval

    Chaorui Deng, Qi Chen, Pengda Qin, Da Chen, and Qi Wu. Prompt switch: Efficient clip adaptation for text-video re- trieval. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15648–15658, 2023. 5, 6, 3

    2023

  7. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 4

    2009

  8. [16]

    Text-guided video masked autoencoder

    David Fan, Jue Wang, Shuai Liao, Zhikang Zhang, Vimal Bhat, and Xinyu Li. Text-guided video masked autoencoder. In European Conference on Computer Vision , pages 282–

  9. [17]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 7

    2019

  10. [18]

    Violet: End-to-end video-language transformers with masked visual-token mod- eling

    Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token mod- eling. arXiv preprint arXiv:2111.12681, 2021. 6

    2021 arXiv

  11. [19]

    Multi-modal transformer for video retrieval

    Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 214–229. Springer, 2020. 5

    2020

  12. [20]

    Openllama: An open repro- duction of llama

    Xinyang Geng and Hao Liu. Openllama: An open repro- duction of llama. https://github.com/openlm- research/open_llama. 3

  13. [21]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  14. [22]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  15. [23]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In European Conference on Computer Vision , pages 87–104. Springer,

  16. [24]

    Video re- cap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Na- garajan, Lorenzo Torresani, and Gedas Bertasius. Video re- cap: Recursive captioning of hour-long videos. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18198–18208, 2024. 2, 3

    2024

  17. [25]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 4

    2021

  18. [26]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Con- ference on Machine Learning, 2021. 1

    2021

  19. [27]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on com- puter vision, pages 706–715, 2017. 5

    2017

  20. [28]

    Video token merging for long-form video under- standing

    Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long-form video under- standing. arXiv preprint arXiv:2410.23782, 2024. 1

    2024 arXiv

  21. [29]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331–7341, 2021. 1...

    2021

  22. [30]

    Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang

    Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length? https://lmsys.org/blog/2023- 06-29-longchat, June 2023. 3

    2023

  23. [31]

    Hero: Hierarchical encoder for video+ lan- guage omni-representation pre-training

    Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ lan- guage omni-representation pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 2046–2065, 2...

    2020

  24. [32]

    Lavender: Unifying video- language understanding as masked language modeling

    Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unifying video- language understanding as masked language modeling. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2023. 6

    2023

  25. [33]

    Ego-exo: Transferring visual representations from third-person to first-person videos

    Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grau- man. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943–6953, 2021. 7

    2021

  26. [34]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 2, 3

    2004

  27. [35]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Sys- tems, 35:7575–7586, 2022. 7

    2022

  28. [36]

    Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement

    Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongx- uan Luo. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10556–10565, 2020. 4

    2021

  29. [37]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016. 5, 7

    2016

  30. [38]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2018. 5, 7

    2018

  31. [39]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neu- rocomputing, 508:293–304, 2022. 1, 2, 5, 6, 3

    2022

  32. [40]

    X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 638–647, 2022. 2, 3, 5, 6

    2022

  33. [41]

    End-to-end learning of visual representations from uncurated instruc- tional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instruc- tional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages...

    2020

  34. [42]

    HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV,

  35. [43]

    Vishal Monga, Yuelong Li, and Yonina C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine , 38:18–44, 2019. 4

    2019

  36. [44]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In European Conference on Computer Vision, pages 529–544. Springer, 2022. 4

    2022

  37. [45]

    Keeping your eye on the ball: Tra- jectory attention in video transformers

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Tra- jectory attention in video transformers. Advances in neural information processing systems, 34:12493–12506...

    2021

  38. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  39. [47]

    Movie description

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 123:94–120, 2017. 1, 5

    2017

  40. [48]

    Actor and observer: Joint modeling of first and third-person videos

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 7396–7404, 2018. 7

    2018

  41. [49]

    Charades-ego: A large-scale dataset of paired third and first person videos

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018. 1, 6

    2018 arXiv

  42. [50]

    Videobert: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464–7473, 2019. 7

    2019

  43. [51]

    Long-form video-language pre- training with multimodal temporal contrastive learning

    Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre- training with multimodal temporal contrastive learning. Ad- vances in neural information processing systems, 35:38032– 38045, 2022. 1, 2, 6, 7, 4

    2022

  44. [52]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1207– 12...

    2019

  45. [53]

    Perceiver-vl: Efficient vision-and-language modeling with iterative latent attention

    Zineng Tang, Jaemin Cho, Jie Lei, and Mohit Bansal. Perceiver-vl: Efficient vision-and-language modeling with iterative latent attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4410–4420, 2023. 2, 5, 6

    2023

  46. [54]

    Internlm: A multilingual language model with progressively enhanced capabilities

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https : / / github.com/InternLM/InternLM, 2023. 3

    2023

  47. [55]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 4

    2016

  48. [56]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 4

    2017

  49. [58]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6387–6397, 2023. 7

    2023

  50. [59]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023. 1, 3, 5, 6, 4

    2023 arXiv

  51. [60]

    Dhp benchmark: Are llms good nlg evaluators? arXiv preprint arXiv:2408.13704, 2024

    Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, and Xia Hu. Dhp benchmark: Are llms good nlg evaluators? arXiv preprint arXiv:2408.13704, 2024. 5

    2024 arXiv

  52. [61]

    Unified coarse-to-fine alignment for video-text retrieval

    Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 2816–2827,

  53. [62]

    Towards long-form video understanding

    Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1884–1894, 2021. 1, 6, 7

    2021

  54. [63]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 7

    2021 arXiv

  55. [64]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 1, 5

    2016

  56. [65]

    Ad- vancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Ad- vancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  57. [66]

    Clip-vip: Adapting pre- trained image-text model to video-language alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language alignment. In The Eleventh International Conference on Learning Repre- sentations, 2023. 5, 6

    2023

  58. [67]

    Just ask: Learning to answer ques- tions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer ques- tions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1686–1697, 2021. 7

    2021

  59. [68]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124–141, 2022. 7

    2022

  60. [69]

    Taco: Token-aware cascade contrastive learning for video-text alignment

    Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11562–11572, 2021. 1, 2, 6, 4

    2021

  61. [70]

    Scaling white-box transform- ers for vision

    Jinrui Yang, Xianhang Li, Druv Pai, Yuyin Zhou, Yi Ma, Yaodong Yu, and Cihang Xie. Scaling white-box transform- ers for vision. arXiv preprint arXiv:2405.20299, 2024. 4

    2024 arXiv

  62. [71]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Rep- resentations, 2021. 4

    2021

  63. [72]

    Hitea: Hierarchical temporal- aware video-language pre-training

    Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. Hitea: Hierarchical temporal- aware video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15405–15416, 2023. 1, 2, 3, 6, 4

    2023

  64. [73]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. 7

    2024

  65. [74]

    Learning from inside: Self- driven siamese sampling and reasoning for video question answering

    Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan. Learning from inside: Self- driven siamese sampling and reasoning for video question answering. Advances in Neural Information Processing Sys- tems, 34:26462–26474, 2021. 7

    2021

  66. [75]

    White-box transformers via sparse rate reduction

    Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems , 36:9422–9457,

  67. [76]

    Bertscore: Evaluating text genera- tion with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert. arXiv preprint arXiv:1904.09675 , 2019. 2, 3

    1904 arXiv

  68. [77]

    Videoprism: A foundational visual encoder for video understanding

    Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In Forty-first International Conference on Machine Learning . 6, 4

  69. [78]

    Cen- terclip: Token clustering for efficient text-video retrieval

    Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Cen- terclip: Token clustering for efficient text-video retrieval. In Proceedings of the 45th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pages 970–981, 2022. 6, 3

    2022

  70. [79]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 3

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.