Pith. sign in

REVIEW 5 major objections 7 minor 72 references

Punching Bag vs. Punching Person: Motion Transferability in Videos

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current action recognition models lose 20 or more accuracy points when a motion appears in a new context, even when the coarse action stays the same.

desk verdict The drop-on-unknown-context finding is solid and worth publishing, but "motion transferability" overstates it for K400-TA, where several coarse groups share scenarios, not motions. read the letter →

arxiv 2508.00085 v1 pith:6R2UESF6 submitted 2025-07-31 cs.CV cs.AI

Raiyaan Abdullah , Jared Claypoole , Michael Cogswell , Ajay Divakaran , Yogesh Rawat This is my paper · ORCID
classification cs.CVcs.AI
keywords actionrecognitionmotiontransferabilityfine-grainedactionscoarse-to-finehierarchyvideounderstandingzero-shotgeneralizationsyntheticbenchmarkdisentanglement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that today's action recognition models learn contexts rather than motions: a model trained on videos of a punching bag does not transfer the broad action 'punching' to an unseen variant such as a person boxing. To test this, the authors build three datasets with a coarse-to-fine hierarchy (Syn-TA, K400-TA, SSv2-TA), split each into two sets with the same coarse actions but disjoint fine classes, and train 13 unimodal and multimodal models to recognize coarse actions in known versus unknown fine contexts. On every dataset and nearly every model, known-to-unknown accuracy drops by 20 points or more on average, and multimodal models drop even more on unknown fine actions. A bias-free synthetic dataset with 3D objects turns out to be the hardest, indicating that models lean on object and background cues rather than on motion itself. The paper also shows that feeding fine-grained context into a coarse branch during training improves coarse action transfer on temporally demanding datasets.

What carries the argument

The load-bearing machinery is the intra-dataset coarse-to-fine evaluation protocol: fine classes that share a coarse motion are split into Set 1 and Set 2, models train on one set and are tested on both known and unknown variants, forcing any correct coarse-action recognition to generalize across disjoint fine contexts. The second piece is a disentanglement head added to EZ-CLIP, duplicating the final transformer blocks into a coarse branch and a fine branch, with fine embeddings added residually to the coarse branch so the model is trained to predict both levels at once. The metrics are the absolute drop $D_{\text{abs}}$ and the harmonic mean between known and unknown accuracies.

What would settle it

Inspect the training corpora of CLIP or the video-language models for the exact unknown fine labels (for example, 'punching person (boxing)', 'cleaning gutters', 'poking a hole into something soft'). If those strings occur in pretraining text and removing or masking them during fine-tuning does not change the reported unknown-fine accuracy, the paper's conclusion that models genuinely understand the motion would be called into question.

Watch

Extended reading notes

Core claim

The central claim is that high-level motion concepts do not transfer across contexts even within a single dataset. The authors formalize this with an intra-dataset coarse-to-fine hierarchy and find that every evaluated model shows a large absolute drop $D_{\text{abs}}$ from known to unknown contexts, regardless of architecture, modality, or domain. They show that multimodal models with frozen CLIP weights sometimes do better on unknown fine classes than on the matching coarse classes because they can latch onto object words in fine captions. They also find that model size helps when spatial appearance dominates (K400-TA) but not when temporal reasoning dominates (SSv2-TA), and that removing background texture in Syn-TA improves results, supporting the context-reliance conclusion. Finally, they claim that a two-branch disentanglement head, with fine features injected into the coarse branch, improves coarse transferability on temporal datasets.

Load-bearing premise

The result depends on the assumption that the fine labels used as 'unknown' classes, such as 'punching person (boxing)', were not already memorized by the pretrained text encoders used in the multimodal models.

Editorial extensions

If this is right

  • On standard benchmarks, reported accuracy overstates motion understanding, because models can solve familiar fine contexts without learning the underlying coarse motion.
  • Model scaling is not a cure for temporal transfer: increasing parameters helped spatial-cue datasets but not Something-Something-v2, so progress on motion transfer needs different architectural or training interventions.
  • Multimodal models that exploit object words in fine captions can appear to understand fine actions while failing the same coarse action, so fine-grained zero-shot numbers need context-normalized interpretation.
  • Using the Syn-TA realistic-versus-plain-background comparison, any future action model can be quickly diagnosed for background dependence before expensive video curation.
  • Training with a coarse-plus-fine objective, as the disentanglement head does, is a concrete recipe for improving coarse transfer on temporally difficult datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical consequence the authors do not spell out: deployment of action recognizers in novel environments (robots, surveillance, assistive AI) should expect large reliability gaps even when the activity category was seen during training.
  • The memorization caveat could be tested directly: search the web-scale text used to pretrain CLIP-style models for the exact fine labels used as unknown classes; if those strings appear, the fine-motion unknown results partly measure label recall rather than transfer.
  • The disentanglement result suggests a cheap intervention for existing video-language models: add a fine-grained auxiliary prediction head and keep it during inference, rather than fine-tuning with coarse labels only.
  • A natural extension the paper does not pursue is to use Syn-TA's controlled generation to vary one factor at a time (object shape, background, camera path, frame order) and pinpoint which manipulation causes the transfer collapse.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces a benchmark framework for evaluating motion transferability in video action recognition. It constructs three datasets—Syn-TA (synthetic, controlled), K400-TA, and SSv2-TA—each organized into a coarse-to-fine hierarchy, and splits the fine classes into two sets that share the same coarse actions but have disjoint fine contexts. The authors train uni-modal and multi-modal models on one set and evaluate on both known and unknown contexts, reporting large accuracy drops (typically Dabs of 20% or more) for coarse actions and even larger drops for fine actions. They further analyze how model size and background texture affect transferability, and propose a disentanglement strategy that adds a fine-motion branch to EZ-CLIP to improve coarse-motion transfer on temporal datasets. The central claim is that current action recognition models are fundamentally context-bound rather than motion-bound.

Significance. If the findings are valid, this work provides a useful diagnostic benchmark for a subtle and under-explored form of generalization: transferring a known high-level motion to a novel fine-grained context within the same dataset. The paper is strong in scope—it evaluates 13+ models on three datasets, provides full class lists and per-set results in the supplementary, and releases code and datasets. The synthetic Syn-TA dataset is a valuable controlled testbed that partially isolates motion from real-world scene biases. However, the central interpretation is currently overextended: several coarse classes are not pure motion concepts, the Syn-TA protocol changes multiple factors simultaneously, and the fine-motion evaluation may be contaminated by pretraining label leakage. These issues do not destroy the empirical measurement of a performance drop, but they do undermine the specific attribution of that drop to failure of motion transfer.

major comments (5)
  1. [§3.2, §D.2 (K400-TA class lists)] The coarse classes in K400-TA are not consistently high-level motions. For example, the group 'toddler interaction' contains 'baby waking up', 'carrying baby', and 'crawling baby'; 'music without instrument' contains 'air drumming', 'beatboxing', and 'singing'; 'gardening' contains 'arranging flowers', 'planting trees', and 'stomping grapes'. In these groups the fine classes do not share a common motion—they share an object or scenario. For such classes, the reported accuracy drop from Set 1 to Set 2 measures failure to generalize object/scene cues, not failure to transfer a motion concept. This directly weakens the central claim in §4.2 that 'all models ... fail to transfer high-level motion concepts to novel contexts.' The authors should restrict the benchmark to coarse groups whose members share a verb or genuine motion pattern, or analyze motion-pure and thematic groups separately.
  2. [§3.2 (Syn-TA construction)] In Syn-TA, the two sets differ simultaneously in object shapes, background images, and fine-motion details ('Each of the two sets has unique background images and object shapes'). Consequently Dabs conflates low-level appearance changes with motion-transfer failure. The statement in §4.2 that Syn-TA 'purely tests a model's motion understanding' is not supported, because a model could fail due to unseen colors, shapes, or textures rather than due to an inability to transfer the motion concept. A cleaner protocol would vary only the fine-motion detail while keeping object geometry and background fixed, or would factorially manipulate each factor to separate its contribution.
  3. [§3.3, Tables 3 and 9] The 'unknown fine motions' for multimodal models include labels such as 'punching person (boxing)' from K400-TA Set 2. These labels are drawn from Kinetics-400, whose class names are very likely present in the web-scale text corpora used to pretrain CLIP and its video variants. The paper does not check for such overlap or exclude memorized labels. If these text descriptions were already seen during pretraining, the high Dabs for FineMotion-Unknown measures recall of memorized text rather than transfer of motion understanding. The authors should verify the absence of the test labels from pretraining data or design a truly novel set of textual descriptions for the unknown fine classes.
  4. [§4.1, Tables 2–11] All experiments are reported as single runs without confidence intervals, error bars, or repeated seeds. The headline numbers—'typical Dabs of 20% or more', the model-size trends in Fig. 6, and the disentanglement gains in Table 5—are point estimates that could shift with random seed, training epochs, or hardware. This is particularly concerning for the disentanglement results, where some gains are tiny (e.g., K400-TA Set 1 HM 76.07 vs. 76.06 in Table 12) and could easily be within run-to-run variance. The authors should provide variance estimates over at least three seeds for the main tables and figures.
  5. [§4.2, Table 2] The cross-dataset comparison of harmonic means is confounded by the different number of coarse classes: Syn-TA has 20, K400-TA has 41, and SSv2-TA has 26. Chance-level accuracy differs substantially across these datasets, so the statement that 'Syn-TA proves as challenging as real-world datasets' is not supported by the raw HM values. A chance-normalized metric (e.g., normalized accuracy) or an analysis with matched class counts is needed before drawing conclusions about the relative difficulty of the synthetic versus real datasets.
minor comments (7)
  1. [Abstract and §4] The paper consistently says '13 state-of-the-art models' in the abstract and introduction, but §4 evaluates 8 unimodal, 5 multimodal, and 3 domain-generalization methods—16 total. Please clarify whether the DG methods are counted separately.
  2. [Table 1] The 'Set 1/Set 2' columns are ambiguous for the coarse-class row; please explicitly show that both sets have the same number of coarse classes, or use separate columns for Set 1 and Set 2.
  3. [§4.2, Fig. 6 and Fig. 12] The bubble-size plots use model parameter counts, but no parameter table is provided. Please include a table listing the number of parameters for each model to make the size analysis reproducible.
  4. [§3.4] Dabs and HM are both reported, but Drel (relative drop) appears only in the supplementary. Since Drel is more interpretable for percentage-based claims, please consider reporting it in the main tables or at least in the main text discussion.
  5. [References] Reference [55] is cited to support the claim that humans can recognize 'punching' beyond 'punching bag', but the cited book appears to be about curriculum design rather than human action generalization. Please replace it with a more relevant citation from the cognitive or action-recognition literature.
  6. [§3.5, Fig. 14] The disentanglement architecture is described only in the supplementary; the main text should include at least the key details—the branching structure, the residual connection formula, and the training loss—since this is presented as one of the paper's contributions.
  7. [Figure 10 caption] The caption refers to 'top' and 'bottom' rows, but the figure appears to have only two panels arranged left and right; please clarify the layout description.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark study whose claims are self-contained measurements.

full rationale

The paper's contributions are dataset construction, a train/eval protocol, and measurements of 13 models under that protocol. The central claim—models drop substantially in coarse accuracy from known to unknown contexts—follows directly from the reported accuracy numbers in Tables 2, 6, 8, and 10, and is not derived from any fitted parameter, self-citation, or definitional identity. The benchmark protocol defines coarse classes by grouping fine classes, but the known/unknown split is a disjoint partition of fine classes, not a quantity fitted to the outcome; the reported Dabs is computed from independent held-out test sets. The disentanglement method in Section 3.5 is tested on the same benchmarks it is designed to improve, which is standard practice for a proposed method and does not make the benchmark's empirical findings circular. The paper does cite prior work by overlapping authors, notably EZ-CLIP [1] used as the base model for the disentanglement experiments, but that citation is not load-bearing for the paper's central transferability measurements; the method's improvement is empirically evaluated rather than assumed. The conceptual concern raised by the skeptic—that some K400-TA coarse groups mix object/scene categories with motions, so the drop may partly reflect context generalization rather than pure motion transfer—is a validity or interpretability question about the benchmark, not a circularity in the derivation, and is therefore outside the scope of this pass. No circular step satisfying the required evidence standard was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities or physical mechanisms are posited. The paper introduces new datasets, but they are not entities in the sense of new forces or particles. The load-bearing assumptions are the validity of manual class grouping, the absence of pretraining label leakage, and the representativeness of the synthetic Syn-TA environment.

assumptions (3)
  • domain assumption The manually curated grouping of fine classes into coarse classes is semantically valid and produces comparable difficulty across Set 1 and Set 2.
    Section 3.1: the validity of the benchmark hinges on this grouping; if it is arbitrary, the measured drop may reflect label difficulty differences rather than transfer failure.
  • domain assumption The unseen fine-class labels are not memorized by the multimodal models during pretraining.
    Section 3.3: CLIP-based models are pretrained on web-scale image-text pairs; labels like 'punching person (boxing)' could appear in pretraining, confounding the unknown fine-motion evaluation.
  • domain assumption Syn-TA's synthetic 3D-object motions fairly represent real-world coarse motions, so failures on Syn-TA indicate motion-understanding deficits rather than synthetic-domain artifacts.
    Section 3.2: the paper's 'bias-free' claim assumes the synthetic distribution does not introduce its own systematic bias that changes the difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Punching Bag vs. Punching Person: Motion Transferability in Videos." pith.science (2026). https://pith.science/paper/6R2UESF6

@misc{pith2026250800085,
  author       = {Pith},
  title        = {Pith review of: Punching Bag vs. Punching Person: Motion Transferability in Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6R2UESF6}},
  note         = {Machine review of arXiv:2508.00085}
}
read the original abstract

Action recognition models demonstrate strong generalization, but can they effectively transfer high-level motion concepts across diverse contexts, even within similar distributions? For example, can a model recognize the broad action "punching" when presented with an unseen variation such as "punching person"? To explore this, we introduce a motion transferability framework with three datasets: (1) Syn-TA, a synthetic dataset with 3D object motions; (2) Kinetics400-TA; and (3) Something-Something-v2-TA, both adapted from natural video datasets. We evaluate 13 state-of-the-art models on these benchmarks and observe a significant drop in performance when recognizing high-level actions in novel contexts. Our analysis reveals: 1) Multimodal models struggle more with fine-grained unknown actions than with coarse ones; 2) The bias-free Syn-TA proves as challenging as real-world datasets, with models showing greater performance drops in controlled settings; 3) Larger models improve transferability when spatial cues dominate but struggle with intensive temporal reasoning, while reliance on object and background cues hinders generalization. We further explore how disentangling coarse and fine motions can improve recognition in temporally challenging datasets. We believe this study establishes a crucial benchmark for assessing motion transferability in action recognition. Datasets and relevant code: https://github.com/raiyaan-abdullah/Motion-Transfer.

Figures

Figures reproduced from arXiv: 2508.00085 by the authors.

Figure 1
Figure 1. Overview of motion transferability: (left) Models fail to transfer high-level coarse motion understanding (‘Punching’) to unknown context not seen during training. State-of-the-art multimodal models like ViFi-CLIP fail to understand unknown fine motions such as (‘Punching person - boxing’) as well. (right) Average detection accuracy (across three datasets) illustrating the performance gap between CoarseMotion-KnownC… view at source ↗
Figure 2
Figure 2. Comparison with zero-shot setup: We categorize mo￾tions into high-level (coarse) groups, creating two sets with similar coarse motions but varied fine details. After training on Train Set 1 (or Set 2), we evaluate performance on known motions in Test Set 1 (or Set 2) and unseen variations in Test Set 2 (or Set 1). deeper model understanding by tackling issues such as ro￾bustness, low-resolution, noisy labels, multi-… view at source ↗
Figure 3
Figure 3. Preview of benchmark datasets: Each dataset is divided into Set 1 and Set 2, where both share similar coarse motions but differ in fine motions. Examples of coarse and fine classes from Syn-TA, K400-TA, and SSv2-TA are provided. main generalization introduces scenario biases. Instead, we systematically group fine-context motions into broader cat￾egories, focusing on which context defines the action rather than locat… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Unimodal vs. multimodal models: Evaluation of fine motions in unimodal models is not possible due to a fixed number of classification heads (left). Multimodal models avoid this limitation since visual and text embeddings both have dimension D (right). and DT estS2 (Coa…
Figure 5
Figure 5. Figure 5: Comparison for multimodal models: Left: Syn-TA, middle: K400-TA, and right: SSv2-TA. Across all datasets, a noticeable performance drop occurs for known to unknown fine motions (red to yellow), similar to the decline in coarse accuracy (blue to green) [PITH_FULL_IMAGE…
Figure 6
Figure 6. Figure 6: Effect of model size on performance: Average harmonic mean of coarse accuracy vs. relative drop Drel. Shape size indi￾cates model scale, and colors distinguish specific models [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Example of failure cases for CoarseMotion-UC in Syn-TA: Models misinterpret coarse motion by failing to analyze all frames, understand their sequence, or track object appearance changes, leading to motion misprediction. maps in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Examples where our approach improves the base model: (left) Syn-TA: The base model misclassifies a single object’s action as “Merging”, failing to recognize the solitary presence. In another instance, it confuses “Merging” with “Cloning”, not discerning the reversed se…
Figure 10
Figure 10. Figure 10: A closer look at “Merging”: The base model (top) ignores some objects before merging, thus mispredicting action as “Cloning”. With disentanglement and fine cues aiding coarse detection, our approach (bottom) focuses on all objects and accu￾rately classifies “Merging”.…
Figure 11
Figure 11. Figure 11: Left: Syn-TA, middle: K400-TA, and right: SSv2-TA. Average detection scores of both sets are given for the three datasets. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Effect of model size for each dataset: Average harmonic mean of coarse accuracy vs. relative drop Drel. Bubble sizes correspond to the total number of model parameters, with colors indicating architecture types (red: CNN, blue: transformer - unimodal, green: transform…
Figure 13
Figure 13. Figure 13: Example of failure cases for FineMotion-U in Syn-TA with ViFi-CLIP, EZ-CLIP: (Top) For the arrival of single objects among many objects, ViFi-CLIP is confusing the motion with occlusion as the arriving pink torus temporarily occludes the objects before completing its …
Figure 14
Figure 14. Figure 14: Disentanglement of coarse and fine video features: In the final two layers of the vision transformer, two branches extract coarse and fine motions simultaneously. The fine embeddings are added to the high-level embedding at each step via residual connections, combinin…
Figure 15
Figure 15. Figure 15: Example cases of K400-TA where unknown fine per￾formance is higher: The bars with a darker color denote coarse classes while their corresponding fine class is shown in a lighter color. The accuracy of coarse classes is noticably lower than fine counterparts. Results a…
Figure 16
Figure 16. Figure 16: Preview of Syn-TA realistic vs. solid plain back￾ground: The absence of complex textures in the background en￾hances the model’s ability to comprehend object motion. This suggests that even for relatively simple motions, models are of￾ten challenged by textured backgr…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 70 canonical work pages

  1. [1]

    Ez- clip: Efficient zeroshot video action recognition, 2024

    Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. Ez- clip: Efficient zeroshot video action recognition, 2024. 1, 2, 5, 6, 7, 3

    work page 2024

  2. [2]

    T2l: Efficient zero-shot action recognition with temporal to- ken learning

    Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. T2l: Efficient zero-shot action recognition with temporal to- ken learning. Transactions on Machine Learning Research,

    work page

  3. [3]

    Are visual- language models effective in action recognition? a compara- tive study, 2024

    Mahmoud Ali, Di Yang, and Franc ¸ois Br´emond. Are visual- language models effective in action recognition? a compara- tive study, 2024. 3

    work page 2024

  4. [4]

    Understanding depth and height percep- tion in large visual-language models

    Shehreen Azad, Yash Jain, Rishit Garg, Vibhav Vineet, and Yogesh Rawat. Understanding depth and height percep- tion in large visual-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3611–3620, 2025. 3

    work page 2025

  5. [5]

    Hierarq: Task-aware hierarchical q-former for enhanced video understanding

    Shehreen Azad, Vibhav Vineet, and Yogesh Singh Rawat. Hierarq: Task-aware hierarchical q-former for enhanced video understanding. In Proceedings of the Computer Vi- sion and Pattern Recognition Conference, pages 8545–8556,

    work page

  6. [6]

    Rethinking zero-shot video classi- fication: End-to-end training for realistic applications, 2020

    Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classi- fication: End-to-end training for realistic applications, 2020. 2

    work page 2020

  7. [7]

    Temporal attentive alignment for large-scale video domain adaptation

    Min-Hung Chen, Zsolt Kira, Ghassan Alregib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6320–6329, 2019. 3

    work page 2019

  8. [8]

    Elaborative rehearsal for zero-shot action recognition, 2021

    Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition, 2021. 2

    work page 2021

Show all 72 references
  1. [9]

    Unsupervised and semi-supervised domain adaptation for action recognition from drones

    Jinwoo Choi, Gaurav Sharma, Manmohan Chandraker, and Jia-Bin Huang. Unsupervised and semi-supervised domain adaptation for action recognition from drones. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1706–1715, 2020. 3

    2020

  2. [10]

    Blender: A 3d modelling and rendering package

    Blender Online Community. Blender: A 3d modelling and rendering package. 2, 4, 6

  3. [11]

    Toyota smarthome: Real-world activities of daily living

    Srijan Das, Rui Dai, Michal Koperski, Luca Minci- ullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. In The IEEE International Conference on Computer Vision (ICCV), 2019. 3

    2019

  4. [12]

    Tinyvirat: Low-resolution video action recognition

    Ugur Demir, Yogesh S Rawat, and Mubarak Shah. Tinyvirat: Low-resolution video action recognition. In 2020 25th Inter- national Conference on Pattern Recognition (ICPR) , pages 7387–7394, 2021. 3

    2020

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Video- capsulenet: a simplified network for action detection

    Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Video- capsulenet: a simplified network for action detection. InPro- ceedings of the 32nd International Conference on Neural In- formation Processing Systems, page 7621–7630, Red Hook, NY , USA, 2018. Curran Associates Inc. 2

    2018

  7. [15]

    Pyslowfast

    Haoqi Fan, Yanghao Li, Bo Xiong, Wan-Yen Lo, and Christoph Feichtenhofer. Pyslowfast. https://github. com/facebookresearch/slowfast, 2020. 1

    2020

  8. [16]

    Multiscale vision transformers, 2021

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers, 2021. 2

    2021

  9. [17]

    X3d: Expanding architectures for efficient video recognition, 2020

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition, 2020. 5, 7, 1, 2, 3

    2020

  10. [18]

    Slowfast networks for video recognition, 2019

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition, 2019. 5, 7, 1, 2, 3

    2019

  11. [19]

    Telling stories for common sense zero-shot action recognition, 2024

    Shreyank N Gowda and Laura Sevilla-Lara. Telling stories for common sense zero-shot action recognition, 2024. 2

    2024

  12. [20]

    A new split for evaluating true zero-shot action recognition, 2021

    Shreyank N Gowda, Laura Sevilla-Lara, Kiyoon Kim, Frank Keller, and Marcus Rohrbach. A new split for evaluating true zero-shot action recognition, 2021. 2

    2021

  13. [21]

    The ”something something” video database for learning and evaluating visual common sense,

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzy ´nska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video d...

  14. [22]

    Hierar- chical explanations for video action recognition, 2023

    Sadaf Gulshad, Teng Long, and Nanne van Noord. Hierar- chical explanations for video action recognition, 2023. 3, 4

    2023

  15. [23]

    Learn- ing spatio-temporal features with 3d residual networks for action recognition, 2017

    Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learn- ing spatio-temporal features with 3d residual networks for action recognition, 2017. 5, 7, 1, 2, 3

    2017

  16. [24]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. 5, 7, 1, 2, 3

    2015

  17. [25]

    Froster: Frozen clip is a strong teacher for open-vocabulary action recognition, 2024

    Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is a strong teacher for open-vocabulary action recognition, 2024. 1, 2, 5, 6, 7, 3

    2024

  18. [26]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. 2, 4, 6

    2017

  19. [27]

    Reformulating zero-shot action recognition for multi- label actions

    Alec Kerrigan, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Reformulating zero-shot action recognition for multi- label actions. In Advances in Neural Information Processing Systems, pages 25566–25577. Curran Associates, Inc., 2021. 2

    2021

  20. [28]

    Learning Cross-Modal Contrastive Features for Video Do- main Adaptation

    Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning Cross-Modal Contrastive Features for Video Do- main Adaptation . In 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 13598–13607, Los ...

    2021

  21. [29]

    Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022

    Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning, 2022. 2, 1 9

    2022

  22. [30]

    Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer, 2022

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer, 2022. 5, 6, 7, 1, 2, 3

    2022

  23. [31]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. 2024. 2

    2024

  24. [32]

    Mvitv2: Improved multiscale vision transformers for classification and detection, 2022

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Man- galam, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Mvitv2: Improved multiscale vision transformers for classification and detection, 2022. 2, 5, 7, 1, 3

    2022

  25. [33]

    Cross-modal representation learning for zero- shot action recognition, 2022

    Chung-Ching Lin, Kevin Lin, Linjie Li, Lijuan Wang, and Zicheng Liu. Cross-modal representation learning for zero- shot action recognition, 2022. 2

    2022

  26. [34]

    Diversifying spatial-temporal perception for video domain generalization

    Kun-Yu Lin, Jia-Run Du, Yipeng Gao, Jiaming Zhou, and Wei-Shi Zheng. Diversifying spatial-temporal perception for video domain generalization. In Advances in Neural Infor- mation Processing Systems, pages 56012–56026. Curran As- sociates, Inc., 2023. 3, 5, 6, 1, 2

    2023

  27. [35]

    Crepe: Can vision-language foundation models reason compositionally? arXiv preprint arXiv:2212.07796, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? arXiv preprint arXiv:2212.07796, 2023. 3

    2023 arXiv

  28. [36]

    Reversible vision transformers, 2023

    Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichtenhofer, and Jitendra Malik. Reversible vision transformers, 2023. 5, 7, 1, 2, 3

    2023

  29. [37]

    Something-else: Com- positional action recognition with spatial-temporal interac- tion networks, 2020

    Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Com- positional action recognition with spatial-temporal interac- tion networks, 2020. 3

    2020

  30. [38]

    Video action detection: Analysing limitations and chal- lenges

    Rajat Modi, Aayush Jung Rana, Akash Kumar, Praveen Tirupattur, Shruti Vyas, Yogesh Rawat, and Mubarak Shah. Video action detection: Analysing limitations and chal- lenges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, ...

    2022

  31. [39]

    Verbs in action: Improving verb understanding in video-language models, 2023

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models, 2023. 3

    2023

  32. [40]

    Multi-modal domain adaptation for fine-grained action recognition, 2020

    Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition, 2020. 3

    2020

  33. [41]

    Expanding language-image pretrained models for gen- eral video recognition, 2022

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition, 2022. 1, 2, 5, 6, 7, 3

    2022

  34. [42]

    Object-relation reasoning graph for action recognition

    Yangjun Ou, Li Mi, and Zhenzhong Chen. Object-relation reasoning graph for action recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20101–20110, 2022. 1

    2022

  35. [43]

    Relative norm align- ment for tackling domain shift in deep multi-modal classifi- cation

    Mirco Planamente, Chiara Plizzari, Simone Alberto Peirone, Barbara Caputo, and Andrea Bottino. Relative norm align- ment for tackling domain shift in deep multi-modal classifi- cation. International Journal of Computer Vision, 132:2618– 2638, 2024. 3

    2024

  36. [44]

    What can a cook in italy teach a mechanic in in- dia? action recognition generalisation over scenarios and locations

    Chiara Plizzari, Toby Perrett, Barbara Caputo, and Dima Damen. What can a cook in italy teach a mechanic in in- dia? action recognition generalisation over scenarios and locations. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13610–13620, 2023. 3,...

    2023

  37. [45]

    Haupt- mann

    Yijun Qian, Lijun Yu, Wenhe Liu, and Alexander G. Haupt- mann. Rethinking zero-shot action recognition: Learning from latent atomic actions. In Computer Vision – ECCV 2022, pages 104–120, Cham, 2022. Springer Nature Switzer- land. 2

    2022

  38. [46]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2, 1

    2021

  39. [47]

    Fine-tuned clip models are efficient video learners, 2023

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners, 2023. 1, 2, 5, 6, 7, 3

    2023

  40. [48]

    Towards a fair evaluation of zero-shot action recognition using external data

    Alina Roitberg, Manuel Martinez, Monica Haurilet, and Rainer Stiefelhagen. Towards a fair evaluation of zero-shot action recognition using external data. In Computer Vi- sion – ECCV 2018 Workshops, pages 97–105, Cham, 2019. Springer International Publishing. 2

    2018

  41. [49]

    Probing conceptual understanding of large visual-language models

    Madeline Schiappa, Raiyaan Abdullah, Shehreen Azad, Jared Claypoole, Michael Cogswell, Ajay Divakaran, and Yogesh Rawat. Probing conceptual understanding of large visual-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pag...

    2024

  42. [50]

    A Large-Scale Robustness Analysis of Video Action Recognition Models

    Madeline Chantry Schiappa, Naman Biyani, Prudvi Kam- tam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yo- gesh Rawat. A Large-Scale Robustness Analysis of Video Action Recognition Models . In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pag...

    2023

  43. [51]

    Noisyactions2m: A multimedia dataset for video understanding from noisy la- bels

    Mohit Sharma, Raj Aaryaman Patra, Harshal Desai, Shruti Vyas, Yogesh Rawat, and Rajiv Ratn Shah. Noisyactions2m: A multimedia dataset for video understanding from noisy la- bels. In Proceedings of the 3rd ACM International Confer- ence on Multimedia in Asia, New York, NY , USA...

    2022

  44. [52]

    Learning long-term dependencies for action recognition with a biologically-inspired deep network

    Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, and Tiejun Huang. Learning long-term dependencies for action recognition with a biologically-inspired deep network. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 716–725, 2017. 1

    2017

  45. [53]

    Dvanet: Disentangling view and action features for multi- view action recognition, 2023

    Nyle Siddiqui, Praveen Tirupattur, and Mubarak Shah. Dvanet: Disentangling view and action features for multi- view action recognition, 2023. 3

    2023

  46. [54]

    Spatio-temporal contrastive domain adaptation for action recognition

    Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio-temporal contrastive domain adaptation for action recognition. In 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9782–9790, 2021. 3

    2021

  47. [55]

    Learning That Transfers: Designing Curricu- lum for a Changing World

    Julie Harris Stern, Krista Fantin Ferraro, Kayla Duncan, and Trevor Aleo. Learning That Transfers: Designing Curricu- lum for a Changing World . Corwin, Thousand Oaks CA,

  48. [56]

    Learning spatiotemporal features with 3d convolutional networks, 2015

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks, 2015. 2

    2015

  49. [57]

    Actionclip: A new paradigm for video action recognition, 2021

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition, 2021. 1, 2, 5, 6, 7, 3

    2021

  50. [58]

    An efficient spatio-temporal pyramid transformer for action detection

    Yuetian Weng, Zizheng Pan, Mingfei Han, Xiaojun Chang, and Bohan Zhuang. An efficient spatio-temporal pyramid transformer for action detection. InComputer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23–27, 2022, Proceedings, Part XXXIV , page 358–...

    2022

  51. [59]

    Action recognition using attention-based spatio-temporal vlad net- works and adaptive video sequences optimization

    Zhengkui Weng, Xinmin Li, and Shoujian Xiong. Action recognition using attention-based spatio-temporal vlad net- works and adaptive video sequences optimization. Scientific Reports, 14(1), 2024. 1

    2024

  52. [60]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding, 2021

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding, 2021. 2

    2021

  53. [61]

    Seman- tic embedding space for zero-shot action recognition, 2015

    Xun Xu, Timothy Hospedales, and Shaogang Gong. Seman- tic embedding space for zero-shot action recognition, 2015. 2

    2015

  54. [62]

    Interact before align: Leveraging cross-modal knowledge for domain adaptive action recognition

    Lijin Yang, Yifei Huang, Yusuke Sugano, and Yoichi Sato. Interact before align: Leveraging cross-modal knowledge for domain adaptive action recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14702–14712, 2022. 3

    2022

  55. [63]

    Aim: Adapting image models for efficient video action recognition, 2023

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition, 2023. 2, 5, 6, 7, 1, 3

    2023

  56. [64]

    Yu, and Mingsheng Long

    Zhiyu Yao, Yunbo Wang, Jianmin Wang, Philip S. Yu, and Mingsheng Long. Videodg: Generalizing temporal relations in videos to novel domains, 2021. 3, 5, 6, 1, 2

    2021

  57. [65]

    Action4d: Online action recognition in the crowd and clutter

    Quanzeng You and Hao Jiang. Action4d: Online action recognition in the crowd and clutter. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11849–11858, 2019. 1

    2019

  58. [66]

    Eliciting in-context learning in vision-language models for videos through curated data dis- tributional properties

    Keunwoo Peter Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data dis- tributional properties. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro...

    2024

  59. [67]

    Derpa- nis

    Weiyu Zhang, Menglong Zhu, and Konstantinos G. Derpa- nis. From actemes to action: A strongly-supervised repre- sentation for detailed action understanding. In 2013 IEEE International Conference on Computer Vision, pages 2248– 2255, 2013. 3

    2013

  60. [68]

    Human-object interaction detection via disentangled transformer, 2022

    Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human-object interaction detection via disentangled transformer, 2022. 3

    2022

  61. [69]

    How can objects help action recognition?, 2023

    Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid. How can objects help action recognition?, 2023. 3 11 Punching Bag vs. Punching Person: Motion Transferability in Videos Supplementary Material The supplementary material provides additional infor- mation to complement t...

    2023

  62. [70]

    Among multi- modal models, we experimented with different variations of CLIP [46] designed for activity recognition

    utilizes the the frozen parameters of pre-trained im- age models and trains various adapters. Among multi- modal models, we experimented with different variations of CLIP [46] designed for activity recognition. ActionCLIP

  63. [71]

    pre-train, prompt, and fine-tune

    adapts a “pre-train, prompt, and fine-tune” approach. X-CLIP [41] proposes a cross-frame module and a video specific prompting scheme to adapt pre-trained language image models. ViFi-CLIP [47] shows that simple fine- tuning can achieve similar results to using specific tempora...

  64. [72]

    Pushing”) and the other focuses on fine-grained con- text (e.g., “Pushing something from left to right

    on coarse classes. Implementation details: For training ResNet50, I3D, X3D, MViTv2, Rev-MViT, and SlowFast we utilized the PySlowFast repository [15] from Meta Research. For other models and domain generalization methods, we used the code from their respective GitHub repositor...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.