Pith. sign in

REVIEW 3 major objections 4 minor 66 references

ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ACE fine-tuning stochastically replaces fixed action labels with LLM-generated synonyms and shadow negatives, improving zero-shot classification of unseen procedural actions.

desk verdict Useful fine-tuning recipe with strong ATA/IKEA gains, but the universal SRT claim is contradicted by the paper's own GTEA numbers. read the letter →

arxiv 2411.15628 v1 pith:ERJ4Z4PX submitted 2024-11-23 cs.CV

Reza Ghoddoosian , Nakul Agarwal , Isht Dwivedi , Behzad Darisuh This is my paper · ORCID
classification cs.CV
keywords actionconceptenhancementzero-shotrecognitionproceduralvideossynonymrobustnessvision-languagemodelsfine-tuningsynonymsshadownegatives
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models can recognize actions they were not trained on, but they overfit to the exact wording of action labels and fail when the same procedural step is described by a synonym. This paper proposes Action Concept Enhancement (ACE), a fine-tuning method that continually swaps fixed action labels for LLM-generated synonyms and adds object-matched 'shadow negatives' during training. ACE reports large gains on zero-shot classification of unseen actions in cooking and assembly videos, with harmonic mean accuracy up to 16, 11, and 4 percent above the second-best baselines on ATA, IKEA, and GTEA, while staying competitive on seen actions. The paper also introduces a Synonym Robustness Test that measures mean accuracy and variance over ten randomly chosen synonym sets, and ACE shows higher mean and lower variance than the baselines there. If the method holds, robustness to unseen action wording becomes a trainable property rather than a fixed limitation of pretrained video-language models.

What carries the argument

The central object is the verb synonym tree: each action is decomposed into a verb-object pair, and the verb is expanded into a tree of LLM-generated synonyms, with the parent repeated as a child at every level to preserve semantic continuity. The argument runs on three mechanisms built from these trees: leaf augmentation in the similarity measure (average over a node's synonym children), randomized action synonyms in an auxiliary classification loss ($L_{rand}$) that creates up to $M^C$ label combinations, and shadow negatives that share the true object but pair it with a wrong verb, extending classification to $C+1$ classes. The fixed root-label loss $L_{fixed}$ anchors the concept while the stochastic loss prevents overfitting to any single surface form.

What would settle it

Run ACE with the same training pipeline but replace each LLM synonym with a randomly drawn verb from a general dictionary (keeping the object fixed). If zero-shot robustness on the Synonym Robustness Test stays at the same high level, the gains are due to label noise rather than to the semantic relation encoded in the synonyms; if it collapses, the concept-semantics explanation is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that stochastic replacement of action labels during fine-tuning teaches a video-language model the action concept behind the label, not just the label string. ACE builds a synonym tree for each action verb, where the root is the dataset's annotated verb and descendants are LLM-generated synonyms up to second order. During each training iteration, every video is classified twice: once against the fixed root labels and once against a freshly randomized set of verb synonyms assigned to the same objects. A 'shadow negative' category, which pairs the true object with a wrong verb, is added to force the model to attend to verbs rather than objects. At inference, the model classifies by the average similarity between the video and all synonyms of a candidate action. On ATA, IKEA, and GTEA, this recipe improves unseen-action accuracy and synonym robustness over fixed-label fine-tuning baselines while keeping seen-action performance competitive.

Load-bearing premise

The method assumes the LLM-generated synonyms are semantically valid and contextually appropriate for the procedural domain; if they are poor or off-distribution, the stochastic training signal can mislead the encoders and the reported gains may not transfer.

Editorial extensions

If this is right

  • ACE improves zero-shot classification of unseen procedural actions without changing the model architecture or pretraining data, by changing only the fine-tuning loss.
  • The Synonym Robustness Test provides a reproducible protocol for measuring whether a VLM is invariant to unseen action synonyms, not just to unseen labels.
  • Both the fixed root-label loss and the randomized synonym loss are necessary: removing $L_{rand}$ or leaf augmentation degrades robustness, and removing shadow negatives reintroduces object bias.
  • The approach transfers across cooking (GTEA) and assembly (ATA, IKEA) domains, suggesting it is tied to the training recipe rather than to a single dataset.
  • Using more synonyms does not monotonically help; the quality of the generated synonyms and the number of fine-tuned encoder layers need to be balanced.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because ACE operates purely on the text side of the model, the same stochastic synonym recipe could be applied to other action taxonomies, such as medical procedures or sports, whenever an LLM can produce in-domain synonyms; the paper does not test these settings.
  • Beyond the paper: Table 4 suggests human-curated synonym trees outperform LLM-only trees, so a hybrid pipeline with light human review of generated synonyms may yield further gains than increasing synonym count.
  • Beyond the paper: the shadow-negative design points to a general cure for object bias in video-language fine-tuning; analogous object-matched wrong-verb negatives could be constructed for any verb-heavy classification task outside procedural videos.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ACE, a fine-tuning technique for vision-language models applied to procedural video action classification. ACE builds per-verb synonym trees using GPT-4, then during training stochastically replaces ground-truth action labels with randomly sampled synonyms in an auxiliary classification loss and augments the classifier with object-matched "shadow negatives" that pair the correct object with a wrong verb. The authors evaluate on ATA, IKEA, and GTEA in a base-to-novel zero-shot setting, reporting consistent gains over existing VLM baselines on seen and unseen default labels, as well as a new Synonym Robustness Test (SRT) that measures accuracy/F1 over 10 random synonym sets for unseen actions. They also provide ablations, an analysis of fine-tuned layers, a manual-synonym comparison, and TSNE visualizations of the learned concept spaces.

Significance. If the reported results hold, ACE is a simple and practical recipe for improving VLM robustness to label variation in procedural domains, with potential application to cooking and assembly assistants. The paper's strengths include a clear training-time augmentation scheme, evaluation across three datasets with multiple baselines, component ablations, and a stated intention to release code. The experiments with human-annotated synonyms (Table 4) are a particularly valuable sanity check of the LLM-generated synonym quality. However, the paper's claim of universal improvement in synonym robustness is contradicted by its own GTEA SRT results, where unseen accuracy exactly matches a baseline; the lack of variance estimates for default-label results further weakens the "significantly beats" language. These issues need to be addressed before the central claims can be fully accepted.

major comments (3)
  1. [Section 4.2, Table 2] The GTEA Synonym Robustness Test shows ACE's unseen accuracy as 45.0±16.8, numerically identical to ProcVLR's 45.0±16.9. This directly contradicts the text's statement that ACE "achieves the highest mean acc and F1 while maintaining low std in all datasets" and the claim that ACE "enhances the action concept understanding of ProcVLR consistently on all datasets." The only SRT improvement on GTEA is in F1 (32.4 vs. 28.8), and no significance test is reported. Since GTEA is the only egocentric and cooking dataset in the study, the universal robustness claim is not supported; the authors should either provide statistical evidence for a meaningful difference or revise the claim to reflect the dataset-dependent nature of the gains.
  2. [Section 4.1/4.2, Tables 1 and 2] Default-label zero-shot results are reported as single point estimates without standard deviations or significance tests across cross-validation folds, although the experimental setup states that IKEA uses 5 splits and GTEA uses 4-fold cross-validation. For GTEA, the unseen-accuracy gain over Text4Vis is only 3.4 points (67.2 vs. 63.8), and the F1 is actually lower (41.0 vs. 47.4). Without error bars or a paired test, the claim that ACE "significantly beats the SoTA in zero-shot classification" is not statistically supported, especially on the small GTEA test set. The authors should report per-fold results or at least mean±std for the default-label experiments.
  3. [Section 3.2/4.3, Table 4] The Synonym Robustness Test uses GPT-4-generated synonyms, the same LLM that supplies training-time augmentations. Because ACE explicitly trains the encoder to align with GPT-4's synonym distribution, SRT may overstate concept understanding relative to human-defined synonyms. Table 4 shows that the GTEA SRT accuracy jumps from 45.0 with GPT test trees to 63.6 with manual test trees, and drops to 41.1 when training on manual trees but testing on GPT trees, confirming a strong sensitivity to the synonym source. The paper should explicitly acknowledge this as a limitation of SRT and, if feasible, include a human-evaluated SRT on at least one additional dataset to demonstrate generalization beyond the LLM's distribution.
minor comments (4)
  1. [Section 3.2, Eq. (3)] The notation for the synonym set is confusing: the text defines v+ = Synonyms(v) ∪ {v} and then uses M as the number of children, but the equation averages over M items without clarifying that this includes the parent node. Please define M consistently and state whether the parent replication is counted in M.
  2. [Section 4.1, Implementation Details] The number of synonym children M varies across datasets (2 for IKEA, 9 for GTEA, 11 for ATA) without a stated criterion. Since M is a free parameter that directly affects the amount of augmentation, the authors should justify this choice or show sensitivity to M on more than the ATA dataset.
  3. [Appendix, Tables 7-9] There are typographical errors in the table captions and section title: "GPT-genrated", "Robustnss", and "genrated" should be corrected to "GPT-generated" and "Robustness".
  4. [Section 4.2, LaVila comparison] The comparison with LaVila in Table 6 is acknowledged as not entirely fair due to different pretraining data. It would be helpful to state explicitly in the main text that LaVila's Ego4D pretraining provides an advantage on egocentric GTEA, even though the appendix does mention this.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ACE is an empirical fine-tuning method evaluated on external benchmarks; no predicted quantity reduces to a fitted input or self-citation by construction.

full rationale

The derivation chain is a training objective (Eqs. 1-9) whose ingredients are root labels, LLM-generated synonym trees, and shadow-negative verbs; the reported zero-shot and SRT numbers are measured on held-out novel classes that are never used to fit those ingredients. The synonym trees for novel classes are generated only at inference from the same GPT-4 process, which creates a distributional similarity to training-time synonyms but not an equivalence: Table 4 shows that replacing GPT test synonyms with manual ones raises SRT accuracy (45.0 to 56.3 for the GPT-trained model), so the result is not forced by the training input. The paper's reliance on GPT-4 synonym quality is an assumption about generalization, not a circular step. The only overlap between the paper and its cited inputs is the ATA dataset [20], which is a benchmark source rather than a load-bearing uniqueness claim or a fitted parameter. One internal inconsistency (GTEA SRT mean accuracy is 45.0 for both ours and ProcVLR in Table 2, despite the text claiming the highest mean in all datasets) is a correctness or consistency issue, not circularity. No equation reduces to a fitted value, and no prediction is defined in terms of the method's own outputs. The default-label results and the manual-synonym robustness test provide independent evidence outside the fitted synonym-generation process.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'concept subspace' and 'shadow negatives' are training/evaluation constructs, not independently existing entities.

free parameters (3)
  • Number of synonym children M per dataset = IKEA: 2, GTEA: 9, ATA: 11
    Chosen by hand per dataset; sensitivity analysis (Fig. 3) shows performance varies with M, but no separate validation split is described for selecting these values.
  • Fine-tuned encoder layers = last 3 attention blocks of TimeSformer + final projection layer of text encoder
    Selected based on experiments in Fig. 4; deeper text fine-tuning degrades zero-shot, deeper video fine-tuning sometimes helps, and the trade-off is dataset-dependent.
  • Temperature tau = 0.02
    Set as a fixed hyperparameter in Eq. 3; described as 'adjusted' without a reported tuning procedure.
assumptions (4)
  • domain assumption Procedural actions decompose as verb + object (a = v ⊕ o)
    Section 3.2 states this decomposition without loss of generality; all synonym trees and shadow negatives are built on it. GTEA uses a placeholder 'ingredient' to fit the template.
  • domain assumption GPT-4-generated synonyms are semantically valid and contextually appropriate
    Section 3.2 and the Appendix prompt rely on GPT-4 producing useful synonyms; Table 4 shows manual synonyms outperform GPT ones, so the method is sensitive to this assumption.
  • domain assumption The base-to-novel split with least-frequent-verb classes as novel produces truly unseen actions
    Section 4.1 selects one-third of action classes with the least frequent verbs; the paper only 'increases the chance' that base and novel verb sets are mutually exclusive, so some novel verbs may overlap with base verbs.
  • ad hoc to paper SRT with 10 random synonym sets is a valid measure of concept understanding
    Section 4.1 introduces this metric; it is not validated against human judgments or an external benchmark, though the manual-synonym experiment in Table 4 provides partial support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos." pith.science (2026). https://pith.science/paper/ERJ4Z4PX

@misc{pith2026241115628,
  author       = {Pith},
  title        = {Pith review of: ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERJ4Z4PX}},
  note         = {Machine review of arXiv:2411.15628}
}
read the original abstract

Vision-language models (VLMs) are capable of recognizing unseen actions. However, existing VLMs lack intrinsic understanding of procedural action concepts. Hence, they overfit to fixed labels and are not invariant to unseen action synonyms. To address this, we propose a simple fine-tuning technique, Action Concept Enhancement (ACE), to improve the robustness and concept understanding of VLMs in procedural action classification. ACE continually incorporates augmented action synonyms and negatives in an auxiliary classification loss by stochastically replacing fixed labels during training. This creates new combinations of action labels over the course of fine-tuning and prevents overfitting to fixed action representations. We show the enhanced concept understanding of our VLM, by visualizing the alignment of encoded embeddings of unseen action synonyms in the embedding space. Our experiments on the ATA, IKEA and GTEA datasets demonstrate the efficacy of ACE in domains of cooking and assembly leading to significant improvements in zero-shot action classification while maintaining competitive performance on seen actions.

Figures

Figures reproduced from arXiv: 2411.15628 by the authors.

Figure 1
Figure 1. Illustration of the similarity between video and text rep [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Synonym trees for the action verbs ’fasten’ and ’insert’ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Impact of the quantity of augmented synonyms on mean [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Impact of fine-tuning various layers of video-text en [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: TSNE visualization for synonym embeddings in seen (outlined in green) and unseen (outlined in red) action spaces of IKEA [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 54 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Ht-step: Aligning instructional articles with how-to videos.Advances in Neural Information Processing Systems, 36, 2024

    Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagara- jan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos.Advances in Neural Information Processing Systems, 36, 2024. 2

    work page 2024

  3. [3]

    Exploring synonyms as context in zero-shot action recognition

    Ioannis Alexiou, Tao Xiang, and Shaogang Gong. Exploring synonyms as context in zero-shot action recognition. In2016 IEEE International Conference on Image Processing (ICIP), pages 4190–4194. IEEE, 2016. 3

    work page 2016

  4. [4]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23066–23078, 2023. 2

    work page 2023

  5. [5]

    The ikea asm dataset: Understanding people assem- bling furniture through actions, objects and pose

    Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Camp- bell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould. The ikea asm dataset: Understanding people assem- bling furniture through actions, objects and pose. In Pro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 847–859, 2021. 2, 5

    work page 2021

  6. [6]

    Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021. 2, 5, 9

    work page 2021

  7. [7]

    Rethinking zero-shot video classification: End-to-end training for realistic applications

    Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Per- ona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4613–4623, 2020. 2

    work page 2020

  8. [8]

    Regen: A good generative zero-shot video classifier should be rewarded

    Adrian Bulat, Enrique Sanchez, Brais Martinez, and Geor- gios Tzimiropoulos. Regen: A good generative zero-shot video classifier should be rewarded. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13523–13533, 2023. 2

    work page 2023

Show all 66 references
  1. [9]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 2, 6

    2017

  2. [10]

    Elaborative rehearsal for zero-shot action recognition

    Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13638–13647, 2021. 2

    2021

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    2018 arXiv

  4. [12]

    Teaching structured vision & language concepts to vision & language models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Rameswar Panda, Roei Herzig, Eli Schwartz, Donghyun Kim, Raja Giryes, Rog´erio Schmidt Feris, Shimon Ullman, et al. Teaching structured vision & language concepts to vision & language models. 2023 ieee. In CVF Conference on Computer Vi...

    2023

  5. [13]

    Step- former: Self-supervised step discovery and localization in instructional videos

    Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. Step- former: Self-supervised step discovery and localization in instructional videos. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    2023

  6. [14]

    Zero-shot action recognition in videos: A survey

    Valter Estevam, Helio Pedrini, and David Menotti. Zero-shot action recognition in videos: A survey. Neurocomputing, 439:159–175, 2021. 2

    2021

  7. [15]

    Ms-tcn: Multi-stage tem- poral convolutional network for action segmentation

    Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage tem- poral convolutional network for action segmentation. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3575–3584, 2019. 5

    2019

  8. [16]

    Learning to recognize objects in egocentric activities

    Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011, pages 3281–3288. IEEE, 2011. 2, 5, 9

    2011

  9. [17]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 9

    2019

  10. [18]

    Prego: online mistake detection in procedural ego- centric videos

    Alessandro Flaborea, Guido Maria D’Amely di Melen- dugno, Leonardo Plini, Luca Scofano, Edoardo De Mat- teis, Antonino Furnari, Giovanni Maria Farinella, and Fabio Galasso. Prego: online mistake detection in procedural ego- centric videos. In Proceedings of the IEEE/CVF Confer...

    2024

  11. [19]

    Improving zero-shot gen- eralization and robustness of multi-modal models

    Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lak- shminarayanan, and Jiaping Zhao. Improving zero-shot gen- eralization and robustness of multi-modal models. In Pro- ceedings of the IEEE/CVF conference on computer visio...

    2023

  12. [20]

    Weakly-supervised action segmentation and unseen error detection in anomalous instructional videos

    Reza Ghoddoosian, Isht Dwivedi, Nakul Agarwal, and Be- hzad Dariush. Weakly-supervised action segmentation and unseen error detection in anomalous instructional videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10128–10138, 2023. 1, 2, 5

    2023

  13. [21]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  14. [22]

    Temporal alignment networks for long-term video

    Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2906–2916, 2022. 2

    2022

  15. [23]

    Probing image-language transformers for verb understanding

    Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141, 2021. 2

    2021 arXiv

  16. [24]

    Clover: Towards a unified video-language alignment and fusion model

    Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xi- aoshuai Sun, and Rongrong Ji. Clover: Towards a unified video-language alignment and fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14856–14866, 2023. 2

    2023

  17. [25]

    Fine-grained generalized zero-shot learning via dense attribute-based attention

    Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4483–4493, 2020. 2

    2020

  18. [26]

    Objects2action: Classifying and localiz- ing actions without any video example

    Mihir Jain, Jan C Van Gemert, Thomas Mensink, and Cees GM Snoek. Objects2action: Classifying and localiz- ing actions without any video example. In Proceedings of the IEEE international conference on computer vision, pages 4588–4596, 2015. 2

    2015

  19. [27]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vi- sion, pages 105–124. Springer, 2022. 2

    2022

  20. [28]

    Error detection in egocentric procedural task videos

    Shih-Po Lee, Zijia Lu, Zekun Zhang, Minh Hoai, and Ehsan Elhamifar. Error detection in egocentric procedural task videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18655– 18666, 2024. 1

    2024

  21. [29]

    Align and prompt: Video-and-language pre-training with entity prompts

    Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953–4963, 2022. 2

    2022

  22. [30]

    Cross-modal representation learning for zero- shot action recognition

    Chung-Ching Lin, Kevin Lin, Lijuan Wang, Zicheng Liu, and Linjie Li. Cross-modal representation learning for zero- shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19978–19988, 2022. 2

    2022

  23. [31]

    Match, expand and im- prove: Unsupervised finetuning for zero-shot action recog- nition with language knowledge

    Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Posseg- ger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. Match, expand and im- prove: Unsupervised finetuning for zero-shot action recog- nition with language knowledge. In Proceedings of th...

    2023

  24. [32]

    Learning to recognize procedural activities with distant supervision

    Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 13853–13863, 2...

    2022

  25. [33]

    Out-of-distribution detection for gener- alized zero-shot action recognition

    Devraj Mandal, Sanath Narayan, Sai Kumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, and Ling Shao. Out-of-distribution detection for gener- alized zero-shot action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, ...

    2019

  26. [34]

    Learning to ground instructional articles in videos through narrations

    Effrosyni Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15201–15213,

  27. [35]

    Object priors for classifying and localizing unseen actions

    Pascal Mettes, William Thong, and Cees GM Snoek. Object priors for classifying and localizing unseen actions. Interna- tional Journal of Computer Vision, 129(6):1954–1971, 2021. 2

    1954

  28. [36]

    End-to-end learning of visual representations from uncurated instruc- tional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instruc- tional videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages...

    2020

  29. [37]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips, 2019

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips, 2019. 2, 6, 10

    2019

  30. [38]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 2

    2013 arXiv

  31. [39]

    Verbs in action: Improving verb understanding in video-language models

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 15579–15591, 2023. 2, 3

    2023

  32. [40]

    Spoken moments: Learning joint audio-visual representations from video descriptions

    Mathew Monfort, SouYoung Jin, Alexander Liu, David Har- wath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition...

    2021

  33. [41]

    Zero-shot temporal action detection via vision-language prompting

    Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xi- ang. Zero-shot temporal action detection via vision-language prompting. In European Conference on Computer Vision , pages 681–697. Springer, 2022. 2

    2022

  34. [42]

    Expanding language-image pretrained models for gen- eral video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition. In European Conference on Com- puter Vision, pages 1–18. Springer, 2022. 2

    2022

  35. [43]

    Learning multimodal representations for unseen activities

    AJ Piergiovanni and Michael Ryoo. Learning multimodal representations for unseen activities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 517–526, 2020. 2

    2020

  36. [44]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vis...

    2023

  37. [45]

    What does a platypus look like? generating customized prompts for zero-shot image classification

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15691–15701, 2023. 2

    2023

  38. [46]

    Alignment-uniformity aware representation learning for zero-shot video classifica- tion

    Shi Pu, Kaili Zhao, and Mao Zheng. Alignment-uniformity aware representation learning for zero-shot video classifica- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 19968–19977,

  39. [47]

    Rethinking zero-shot action recognition: Learning from latent atomic actions

    Yijun Qian, Lijun Yu, Wenhe Liu, and Alexander G Haupt- mann. Rethinking zero-shot action recognition: Learning from latent atomic actions. InEuropean Conference on Com- puter Vision, pages 104–120. Springer, 2022. 2

    2022

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  41. [49]

    Language-based action concept spaces improve video self-supervised learn- ing

    Kanchana Ranasinghe and Michael S Ryoo. Language-based action concept spaces improve video self-supervised learn- ing. Advances in Neural Information Processing Systems , 36:74980–74994, 2023. 3

    2023

  42. [50]

    Fine-tuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6545–6554, 2023. 1, 2, 5, 6

    2023

  43. [51]

    Is a caption worth a thousand im- ages? a controlled study for representation learning

    Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth a thousand im- ages? a controlled study for representation learning. arXiv preprint arXiv:2207.07635, 2022. 2

    2022 arXiv

  44. [52]

    As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2022

  45. [53]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857–16867, 2020. 2

    2020

  46. [54]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1207– 12...

    2019

  47. [55]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 2

    2021 arXiv

  48. [56]

    Vilta: Enhancing vision-language pre-training through textual augmentation

    Weihan Wang, Zhen Yang, Bin Xu, Juanzi Li, and Yankui Sun. Vilta: Enhancing vision-language pre-training through textual augmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 3158– 3169, 2023. 2

    2023

  49. [57]

    Pax- ion: Patching action knowledge in video-language founda- tion models

    Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Pax- ion: Patching action knowledge in video-language founda- tion models. Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  50. [58]

    Zero-shot event detection using multi-modal fusion of weakly supervised concepts

    Shuang Wu, Sravanthi Bondugula, Florian Luisier, Xiaodan Zhuang, and Pradeep Natarajan. Zero-shot event detection using multi-modal fusion of weakly supervised concepts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2665–2672, 2014. 2

    2014

  51. [59]

    Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10704–10713, 2023

    Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10704–10713, 2023. 3

    2023

  52. [60]

    Revisiting clas- sifier: Transferring vision-language models for video recog- nition

    Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting clas- sifier: Transferring vision-language models for video recog- nition. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 2847–2855, 2023. 1, 6

    2023

  53. [61]

    Bidirectional cross- modal knowledge exploration for video recognition with pre-trained vision-language models

    Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirectional cross- modal knowledge exploration for video recognition with pre-trained vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,...

    2023

  54. [62]

    Generative action description prompts for skeleton-based action recognition

    Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. Generative action description prompts for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10276–10285, 2023. 3

    2023

  55. [63]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 2

    2021 arXiv

  56. [64]

    Movie genre classification by language augmentation and shot sampling

    Zhongping Zhang, Yiwen Gu, Bryan A Plummer, Xin Miao, Jiayi Liu, and Huayan Wang. Movie genre classification by language augmentation and shot sampling. InProceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 7275–7285, 2024. 2

    2024

  57. [65]

    Learning video representations from large lan- guage models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Girdhar. Learning video representations from large lan- guage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6586– 6597, 2023. 1, 2, 3, 9, 10

    2023

  58. [66]

    Learning procedure-aware video represen- tation from instructional videos and their narrations

    Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure-aware video represen- tation from instructional videos and their narrations. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14825–14835, 20...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.