Pith. sign in

REVIEW 4 major objections 6 minor 44 references

Efficient Transfer Learning for Video-language Foundation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-modal spatio-temporal adapter that trains only 2-7% of the parameters matches or beats full fine-tuning on video action recognition while keeping generalization to unseen classes.

desk verdict Solid incremental PEFT paper for ViCLIP, but the SOTA claim needs a corrected loss equation and a validation protocol that doesn't tune on held-out classes. read the letter →

arxiv 2411.11223 v4 pith:D76DMFSZ submitted 2024-11-18 cs.CV

Haoxing Chen , Zizheng Huang , Yan Hong , Yanshuo Wang , Zhongcai Lyu , Zhuoer Xu , Jun Lan , Zhangxuan Gu This is my paper · ORCID
classification cs.CV
keywords parameter-efficienttransferlearningvideoactionrecognitionvideo-languagefoundationmodelmulti-modaladapterspatio-temporaldescriptionconsistencyconstraintbase-to-novelgeneralizationfew-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pre-trained video-language models can be adapted to action recognition without unfreezing their weights. The paper proposes the Multi-modal Spatio-Temporal Adapter (MSTA), a set of lightweight projection blocks inserted into the top layers of both the text and video encoders, plus a shared projection layer that aligns the two modalities. To keep the model from overfitting to the small training set, a spatio-temporal description-guided consistency constraint forces the trainable text branch to stay close to the frozen branch when both are fed LLM-generated descriptions of the action's visual appearance and temporal steps. The authors report that this setup beats full fine-tuning and prior efficient-transfer methods on zero-shot, few-shot, base-to-novel, and fully-supervised video action recognition while training only 2-7% of the original parameters. If the claim holds, parameter-efficient adapters are not just a compute-saving compromise but a way to improve generalization to unseen classes.

What carries the argument

The load-bearing machinery is the shared projection layer inside MSTA. Instead of independent uni-modal adapters, MSTA routes both text and video features through a common learned projection, so the shared space receives gradients from both modalities and explicitly aligns text and video representations. Around it, each video adapter has a down-projection, a shared middle projection, and two up-projections (spatial linear, temporal 3D convolution); each text adapter has down, shared, and up projections. The second load-bearing mechanism is the consistency loss, which anchors the trainable text branch to the frozen one by requiring template embeddings to stay close to embeddings of LLM-generated descriptions of the action's visual appearance and temporal steps. This acts as a regularizer against overfitting and catastrophic forgetting.

What would settle it

A decisive test is to rerun the base-to-novel protocol on UCF-101 and Something-Something v2 with the number of LLM descriptions set to N=8 and compare with the paper's N=2 result: the paper's own hallucination explanation predicts the harmonic mean should drop, so if it rises or stays flat the claimed mechanism is not the cause of the gains. Equally, replacing the LLM descriptions with random sentence-length noise of the same template should collapse performance if the semantic content is load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that adapting a frozen video-language model with MSTA plus a spatio-temporal description-guided consistency constraint yields better accuracy than full fine-tuning across zero-shot, few-shot, base-to-novel, and fully-supervised video action recognition, with only 2-7% of the parameters trainable. MSTA inserts lightweight down/up projection blocks into the upper layers of both the video and text encoders, shares one projection layer between the two branches so gradients from both modalities pull the representations into a common space, and splits the video up-projection into spatial (linear) and temporal (3D convolution) paths. The consistency constraint feeds class templates to the trainable text branch and LLM-generated spatial and temporal descriptions to the frozen text branch, then penalizes the cosine distance between their embeddings; the total loss is $\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \alpha \mathcal{L}_{\mathrm{CC}}$. On Kinetics-400, HMDB-51, UCF-101, Something-Something v2, and Kinetics-600, the paper reports the best harmonic mean between base and novel classes and the best few-shot and zero-shot accuracies among the compared methods, at 8.7M trainable parameters.

Load-bearing premise

The load-bearing premise is that the generated descriptions are the right semantic anchors; if they misdescribe or fail to distinguish classes, the consistency constraint will pull the trainable branch toward wrong text embeddings and reduce accuracy, which is exactly the degradation the authors observe when they increase the number of descriptions from 2 to 8.

Editorial extensions

If this is right

  • If MSTA holds, full fine-tuning of video-language models becomes unnecessary for action recognition: adapter-only training with 2-7% of parameters gives equal or better accuracy and better generalization to unseen classes.
  • The consistency constraint gives a practical recipe for injecting external knowledge from LLM descriptions into frozen multi-modal encoders without unfreezing them, which should transfer to other video tasks such as temporal action localization or video retrieval.
  • Because the shared projection gets gradients from both modalities, the same adapter design can be expected to improve text-video alignment in any contrastively trained video-language model, not only the ViCLIP backbone used here.
  • In few-shot regimes the paper reports gains of several points on Something-Something v2 (for example, 17.5 versus 15.4 at K=16), suggesting that even very small labeled video sets can be exploited without catastrophic forgetting.
  • Fully-supervised Kinetics-400 results improve from 79.9 top-1 with the frozen backbone to 82.2 with MSTA plus the consistency constraint, showing the adapter also helps when training data is plentiful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the N=2 optimum in the authors' ablation implies description quality matters more than quantity; an automatic description-filtering or ensembling step that discards hallucinated sentences could push base-to-novel performance higher than the reported numbers.
  • Beyond the paper: because the consistency loss only needs paired template and description embeddings, it could be applied with description sources other than LLMs, such as video subtitles or automatically mined caption sets, removing the dependency on a specific external language model.
  • Beyond the paper: a shared projection layer that receives gradients from both modalities suggests the adapter could be reused for cross-modal retrieval or open-vocabulary video classification with the same trained weights, since text-video alignment is the explicit optimization target.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MSTA, a parameter-efficient multi-modal spatio-temporal adapter for transferring video-language foundation models (ViCLIP) to video action recognition. The adapter is inserted into the higher layers of both the text and video encoders, with a shared projection layer for cross-modal alignment and separate spatial/temporal up-projections in the video branch. To preserve generalization, the authors add a spatio-temporal description-guided consistency constraint that enforces agreement between the trainable text branch and a frozen pre-trained text branch fed with LLM-generated action descriptions. The method is evaluated on zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised recognition across several datasets, reporting state-of-the-art results while training only 2-7% of the model parameters. The paper includes extensive ablations on adapter design, hyperparameters, and description counts.

Significance. If the claims are substantiated, this is a useful contribution to efficient transfer learning for video-language models: it demonstrates that a lightweight adapter with a regularization constraint can match or exceed full fine-tuning and existing PEFT baselines on multiple benchmarks. The experimental coverage is broad (six datasets, four evaluation protocols), and the comparisons against AdaptFormer and LoRA on the same ViCLIP backbone are appropriate. The paper also makes an honest observation in Table 6e that LLM hallucination degrades performance as the number of descriptions grows, which tempers the method's dependence on description quality. However, the significance of the claimed state-of-the-art results is currently undercut by (i) an incorrectly specified consistency loss in Eq. (14), (ii) hyperparameter selection using the held-out novel-class harmonic mean, and (iii) the absence of released descriptions and error bars in most tables. These issues are fixable, but until addressed the numerical claims lack independent support.

major comments (4)
  1. [3.3, Eq. (14)] The consistency loss as printed in Eq. (14) is not a valid loss function. The first term has the scalar dot product w_c · D_s in the numerator but a denominator containing ||w_c · D_t||, which is the norm of a vector (or a scalar, depending on interpretation), and the second term w_c / (||w_c|| ||D_t||) is a vector, not a scalar loss. Dimensionally the two terms cannot be combined into a scalar loss. Moreover, the text states that the constraint enforces consistency between the trainable branch and the pre-trained branch, but Eq. (14) only involves quantities attributed to the pre-trained branch (w_c, D_s^c, D_t^c). The actual regularization term therefore cannot be reconstructed from the paper. Please provide a dimensionally consistent, unambiguous definition that specifies which embeddings come from the trainable branch and which from the pre-trained branch, and how the cosine distance is computed. This is load-bearing because the consistency constraint is one of the two central contributions and its benefits are reported in Tables 2-5.
  2. [4.3, Table 6, Table 1] The hyperparameters in Table 1 appear to be selected using the base-to-novel harmonic mean reported in Table 6. Since the harmonic mean includes accuracy on novel classes, which are held out during training, this selection procedure leaks test information into model choice. The final settings (lambda=0.005, alpha=1.0, N=2, Dims=256, Layers=1-12) coincide with the values that maximize HM in Table 6. Given that the improvements over AdaptFormer and LoRA in Table 2 are modest (e.g., HM gains of +0.6 to +1.7), the reported ranking may partly be a tuning artifact. The authors should either tune hyperparameters on a validation split that excludes novel classes, or report the sensitivity of the main comparisons to reasonable hyperparameter variations to show that the conclusions are robust.
  3. [3.3, Table 6e] The spatio-temporal descriptions generated by the LLM are a load-bearing input to the consistency constraint, but the full set of descriptions is not released in the paper or supplementary material. Table 6e shows that performance degrades when N increases from 2 to 8, which the authors attribute to LLM hallucination; this indicates that the benefit of the consistency loss depends heavily on the exact descriptions and the generation procedure. Without releasing the descriptions and specifying the prompt details, LLM version, and decoding settings, the consistency-loss contribution is not reproducible and its claimed effect cannot be independently evaluated.
  4. [Tables 2, 3, 5] Tables 2, 3, and 5 report no standard deviations or multiple-seed runs, while several differences between MSTA+LCC and the strongest PEFT baselines are small (e.g., Table 2 HM on UCF-101: 82.9 vs. 82.3; Table 3 HMDB-51 K=16: 70.4 vs. 70.1). Without error bars or significance tests, it is unclear whether these differences are meaningful. The variance reporting in Table 4 (mean ± std) is a good model and should be extended to the other main tables.
minor comments (6)
  1. [Section 1 vs. 4.1] The introduction lists six benchmark datasets including ActivityNet, but Section 4.1 lists only five datasets and no ActivityNet results are reported anywhere; please reconcile this inconsistency.
  2. [Section 3.3 text after Eq. (14)] The paper refers to the consistency constraint as a 'cosine distance' but Eq. (14) is not the standard cosine distance; please align the terminology with the corrected equation.
  3. [Section 4.2, SSv2 comparison] The sentence 'we improve the recognition accuracy of the novel category on SSv2 from 11.5 to 15.8 (an increase of 30 37%)' contains a garbled fragment '30 37%'; the increase is 37% and '30' should be removed.
  4. [Section 4.3, first paragraph] The reference to 'Table 4a' should be 'Table 6a', since the component-wise ablation is shown in Table 6.
  5. [Eq. (15)] The softmax expression in Eq. (15) has an unmatched parenthesis and is missing a closing brace for the logarithm; please correct the typesetting.
  6. [Author affiliations and footnote] There are spelling errors in the author block: 'Nanjing Univerisy' should be 'Nanjing University' and 'avaliable' should be 'available'. Also, 'A5' in Tables 2 and 3 is not defined; if it refers to AdaptFormer, please use the full name.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: main results are empirical comparisons on held-out splits, and the consistency constraint is a regularizer rather than an encoding of the target metric.

full rationale

The paper's central claims are empirical. MSTA is an adapter architecture plus a consistency regularizer; neither the adapter equations (Eqs. 7-13) nor the loss (Eqs. 14-16) define the reported accuracies. The consistency loss compares the learnable text branch (fed template prompts) with the frozen pre-trained branch (fed LLM-generated spatio-temporal descriptions); the soft targets come from fixed ViCLIP/DeepSeek outputs, not from the evaluation labels, so the constraint is not a self-fulfilling definition of base/novel or zero-shot performance. The main generalization numbers are measured on held-out novel classes (Table 2) and on datasets not used for fine-tuning (Table 4), so they are not fitted values in the prediction sense. The base-to-novel harmonic mean is used in Table 6 to select hyperparameters, which is a potential protocol/leakage concern about tuning on a metric that includes novel classes, but it is a benchmarking fairness issue rather than a circular derivation: the reported numbers are not forced to equal the hyperparameter objective by construction, and the same architecture is also compared on independent zero-shot and fully-supervised benchmarks. The malformed Eq. 14 (vector subtraction and mismatched norms) is a correctness/typo issue that does not make the method circular. The paper's own observation that increasing the description count N hurts performance due to LLM hallucination (Table 6e) is a stated limitation of the description source, not evidence of circularity. Self-citations are limited to related-work context (e.g., ref. [3]) and are not load-bearing for any result. No circularity score above 0 is warranted.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities or conceptual objects. It relies on existing models (ViCLIP, DeepSeek) and hand-set hyperparameters. The free parameters listed above are chosen by ablation on the evaluation sets, which is the main circularity concern.

free parameters (5)
  • scaling factor lambda = 0.005 (0.001 for zero-shot)
    Controls the weight of adapter output; selected by ablation in Table 6c to maximize HM, which includes novel classes.
  • LCC weighting alpha = 1.0
    Weight of the consistency loss in the final loss; chosen by ablation in Table 6d.
  • number of descriptions N = 2
    Number of LLM-generated spatio-temporal descriptions per class; chosen by ablation in Table 6e.
  • adapter hidden dimension = 256 (base-to-novel), 128 (few-shot/zero-shot)
    Dimension of the shared projection layer; set by design and ablated in Table 6b.
  • layer insertion start k = 1 (most tasks), 8 (few-shot)
    The first transformer layer at which adapters are inserted; ablated in Table 6f.
assumptions (4)
  • domain assumption LLM-generated spatio-temporal descriptions are class-discriminative and reliable
    The consistency loss in Sec. 3.3 relies on these descriptions; Table 6e shows performance degrades with N=8 due to LLM hallucination.
  • domain assumption Pre-trained ViCLIP text encoder embeddings of template and descriptive sentences are comparable via cosine similarity
    The consistency loss uses cosine distance between embeddings of template inputs and LLM descriptions without empirical validation of this assumption.
  • domain assumption Frozen ViCLIP provides a suitable base for adaptation
    All experiments build on ViCLIP; no comparisons with other frozen video backbones are provided.
  • standard math Standard backpropagation through adapters with frozen base model yields stable optimization
    Assumed by using AdamW and standard fine-tuning; not proven here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Transfer Learning for Video-language Foundation Models." pith.science (2026). https://pith.science/paper/D76DMFSZ

@misc{pith2026241111223,
  author       = {Pith},
  title        = {Pith review of: Efficient Transfer Learning for Video-language Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D76DMFSZ}},
  note         = {Machine review of arXiv:2411.11223}
}
read the original abstract

Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional modules to capture temporal information. Although the additional modules increase the capacity of model, enabling it to better capture video-specific inductive biases, existing methods typically introduce a substantial number of new parameters and are prone to catastrophic forgetting of previously acquired generalizable knowledge. In this paper, we propose a parameter-efficient Multi-modal Spatio-Temporal Adapter (MSTA) to enhance the alignment between textual and visual representations, achieving a balance between generalizable knowledge and task-specific adaptation. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint.This constraint involves providing template inputs (e.g., "a video of \{\textbf{cls}\}") to the trainable language branch and LLM-generated spatio-temporal descriptions to the pre-trained language branch, enforcing output consistency between the branches. This approach reduces overfitting to downstream tasks and enhances the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model.

Figures

Figures reproduced from arXiv: 2411.11223 by the authors.

Figure 1
Figure 1. We compared the number of trainable parameters of our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed Multi-modal Spatio-Temporal [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 32 canonical work pages

  1. [1]

    Quo vadis, action recognition? A new model and the kinetics dataset

    Jo˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724–4733, 2017. 2, 5, 6, 7

    work page 2017

  2. [2]

    A short note about kinet- ics600

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinet- ics600. arXiv preprint arXiv:1808.01340, 2018. 2, 5

    arXiv 2018

  3. [3]

    Conditional Prototype Rectification Prompt Learning

    Haoxing Chen, Yaohui Li, Zizheng Huang, Yan Hong, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Huijia Zhu, and Weiqiang Wang. Conditional prototype rectification prompt learning. arXiv preprint arXiv:2404.09872, 2024. 1, 3

    work page Pith review arXiv 2024

  4. [4]

    Elaborative rehearsal for zero- shot action recognition

    Shizhe Chen and Dong Huang. Elaborative rehearsal for zero- shot action recognition. In ICCV, pages 13618–13627, 2021. 6, 7

    work page 2021

  5. [5]

    Adaptformer: Adapt- ing vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yib- ing Song, Jue Wang, and Ping Luo. Adaptformer: Adapt- ing vision transformers for scalable visual recognition. In NeurIPS, 2022. 2, 4, 7

    work page 2022

  6. [6]

    OST: refining text knowledge with optimal spatio-temporal descriptor for general video recognition

    Tom Tongjia Chen, Hongshan Yu, Zhengeng Yang, Zechuan Li, Wei Sun, and Chen Chen. OST: refining text knowledge with optimal spatio-temporal descriptor for general video recognition. In CVPR, pages 18888–18898, 2024. 3, 5, 6, 7

    work page 2024

  7. [7]

    Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model

    DeepSeek-AI. Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 2, 5

    arXiv 2024

  8. [8]

    BERT: pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In NAACL, pages 4171– 4186, 2019. 3

    work page 2019

Show all 44 references
  1. [9]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, pages 11573–11582, 2022. 3

    2022

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  3. [11]

    Zero-shot and few-shot video question answering with multi-modal prompts

    Deniz Engin and Yannis Avrithis. Zero-shot and few-shot video question answering with multi-modal prompts. InICCV Workshops, pages 2797–2802, 2023. 3

    2023

  4. [12]

    Promptdet: Towards open-vocabulary detection using uncurated images

    Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In ECCV, pages 701–717, 2022. 1

    2022

  5. [13]

    The ”something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fr¨und, Peter Yianilos, Moritz Mueller- Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video da...

    2017

  6. [14]

    Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. InICCV, pages 1026–1034,

  7. [15]

    Activitynet: A large-scale video bench- mark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video bench- mark for human activity understanding. In CVPR, pages 961–970, 2015. 2

    2015

  8. [16]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799, 2019. 4, 7

    2019

  9. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 2, 4, 7

    2022

  10. [18]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, pages 105–124, 2022. 6, 7

    2022

  11. [19]

    Khan, and Fahad Shahbaz Khan

    Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pages 19113–19122, 2023. 1, 3

    2023

  12. [20]

    Poggio, and Thomas Serre

    Hildegard Kuehne, Hueihan Jhuang, Est ´ıbaliz Garrote, Tomaso A. Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In ICCV, pages 2556–2563, 2011. 2, 5, 6

    2011

  13. [21]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 1, 2

    2023

  14. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5

    2019

  15. [23]

    Timo L¨uddecke and Alexander S. Ecker. Image segmentation using text and image prompts. In CVPR, pages 7076–7086,

  16. [24]

    Expanding language-image pretrained models for general video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In ECCV, pages 1–18, 2022. 2, 3, 5, 6, 7

    2022

  17. [25]

    St-adapter: Parameter-efficient image-to-video transfer learning

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurIPS, 2022. 2

    2022

  18. [26]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  19. [27]

    Haupt- mann

    Yijun Qian, Lijun Yu, Wenhe Liu, and Alexander G. Haupt- mann. Rethinking zero-shot action recognition: Learning from latent atomic actions. In ECCV, pages 104–120, 2022. 6, 7

    2022

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748–...

    2021

  21. [29]

    Khan, and Fahad Shahbaz Khan

    Hanoona Abdul Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Fine-tuned CLIP models are efficient video learners. In CVPR, pages 6545–6554, 2023. 1, 3, 6 9

    2023

  22. [30]

    Consistency-guided prompt learning for vision-language models

    Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In ICLR, 2024. 2

    2024

  23. [31]

    UCF101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 2, 5, 6

    2012 arXiv

  24. [32]

    Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. 6

    2022

  25. [33]

    Repre- sentation learning with contrastive predictive coding

    A¨aron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 2

    2018 arXiv

  26. [34]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 2, 3, 7

    2021 arXiv

  27. [35]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...

    2022 arXiv

  28. [36]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2024. 1, 2, 3, 5

    2024

  29. [37]

    Khan, Fa- had Shahbaz Khan, and Mubarak Shah

    Syed Talal Wasim, Muzammal Naseer, Salman H. Khan, Fa- had Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive CLIP via multimodal prompting. In CVPR, pages 23034–23044, 2023. 2, 3, 6, 7

    2023

  30. [38]

    Open-vclip: Transforming CLIP to an open-vocabulary video model via interpolated weight optimization

    Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming CLIP to an open-vocabulary video model via interpolated weight optimization. In ICML, pages 36978–36989, 2023. 3

    2023

  31. [39]

    CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching

    Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching. In CVPR, pages 7031–

  32. [40]

    MMA: multi-modal adapter for vision-language models

    Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiaohua Xie. MMA: multi-modal adapter for vision-language models. In CVPR, pages 23826–23837, 2024. 2

    2024

  33. [41]

    AIM: adapting image models for efficient video action recognition

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: adapting image models for efficient video action recognition. In ICLR, 2023. 2

    2023

  34. [42]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pe...

    2021 arXiv

  35. [43]

    Tip- adapter: Training-free adaption of clip for few-shot classifica- tion

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classifica- tion. In ECCV, pages 493–510, 2022. 1, 3

    2022

  36. [44]

    Mote: Reconciling generalization with specialization for visual-language to video knowledge transfer

    Minghao Zhu, Zhengpu Wang, Mengxian Hu, Ronghao Dang, Xiao Lin, Xun Zhou, Chengju Liu, and Qijun Chen. Mote: Reconciling generalization with specialization for visual-language to video knowledge transfer. arXiv preprint arXiv:2410.10589, 2024. 2, 3, 6, 7 10

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.