Pith. sign in

REVIEW 4 major objections 5 minor 250 references

This survey argues that the structure of video datasets—motion complexity, temporal span, compositionality, and multimodal richness—is the principal force shaping model architecture, making dataset design a strategic lever for the field.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 19:34 UTC pith:FKHX2A4F

load-bearing objection A useful survey with a serious internal contradiction: its own benchmark table refutes its central claim that early 3D CNNs dominate short-clip datasets. the 4 major comments →

arxiv 2509.09151 v2 pith:FKHX2A4F submitted 2025-09-11 cs.CV cs.AIcs.LG

Video Understanding by Design: How Datasets Shape Video Models

Lei Wang , Syuan-Hao Li , Piotr Koniusz , Yongsheng Gao This is my paper
classification cs.CV cs.AIcs.LG
keywords video understandingdataset-centric analysisinductive biasarchitecture evolutionaction recognitionvideo transformersvision-language modelsbenchmark analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a single organizing claim: video-understanding architectures do not evolve on their own; they are responses to the structural properties of the datasets the field builds and adopts. Four properties—motion amplitude, temporal span, compositional/hierarchical structure, and multimodal richness—act as structural pressures that favor specific inductive biases, from two-stream CNNs and 3D convolutions for short motion clips, to temporal reasoning networks and transformers for long procedural sequences, to vision-language models for text-paired corpora. A sympathetic reader should care because this reframes dataset construction as an active design choice that determines which kinds of video intelligence become possible, and it offers a matching rule: pick the architecture whose inductive bias fits the dataset's structure. The paper supports the claim with a large comparative table of datasets and benchmarks spanning two decades.

Core claim

The central discovery is that datasets operate as inductive-bias generators. Each dataset imposes invariances its model must internalize: coarse high-amplitude motions reward instantaneous motion capture (optical flow, shallow 3D filters); long-horizon, overlapping activities reward temporal memory and hierarchy; multi-agent scenes reward relational or graph representations; and video-text corpora reward cross-modal alignment. On this reading, the milestone trajectory—two-stream networks, 3D CNNs, temporal segment/relation networks, transformers, masked self-supervised models, and video-language foundation models—is not a random succession of fashions but a systematic accommodation of increa

What carries the argument

The carrying mechanism is the dataset-bias-architecture framework. It breaks video datasets into four structural properties—motion amplitude, temporal span, compositionality/hierarchy, and multimodal richness (plus agent density)—and treats every architecture as an enforced inductive bias that matches (or mismatches) those properties. Table II operationalizes the framework as a compact rating system (H/M/L for amplitude and agents, S/M/L for span, -/C/H for composition) across a century of datasets; Tables III and IV connect those ratings to measured performance of representative models. The framework does the explanatory work: it turns the history of video understanding into a sequence of d

Load-bearing premise

The load-bearing premise is that the four hand-selected dataset attributes are the dominant cause of architectural change—rather than compute availability, leaderboard incentives, or model-family trends—and that the paper's H/M/L ratings of each dataset are accurate and sufficient.

What would settle it

Run a controlled study that keeps the architecture family fixed, varies a single dataset attribute (e.g., temporal span while holding motion amplitude constant), and shows no systematic performance ordering; alternatively, find two datasets with identical attribute ratings that produced very different dominant architectures. Either result would break the claimed causal link.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • Matching architecture to dataset structure pays off: short-clip motion datasets favor two-stream and 3D CNN models, compositional and interaction-heavy datasets favor sequential and transformer models, and text-paired corpora favor video-language pretraining.
  • Training on coarse, motion-only datasets yields fragile transfer; if robustness in nuanced real-world settings is desired, motion granularity must appear in the data.
  • Simply scaling class counts or clip counts will not yield general video intelligence; the decisive ingredient is structure—procedural hierarchies, temporal continuity, and precise cross-modal alignment.
  • Future architectures should integrate temporal precision, hierarchical composition, long-horizon attention, and multimodal grounding; future datasets should be built with sub-second audio-text alignment, multi-agent annotations, and compositional evaluation splits.
  • Dataset design should be treated as a strategic lever, not a scaling exercise, because datasets generate the invariance pressures that architectures evolve to accommodate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The causal arrow (datasets shape architectures) could be tested directly by controlled experiments that fix the architecture family and vary one structural attribute at a time; the survey does not run such ablations, so the claim remains an interpretation of correlated historical patterns.
  • Editorial inference: If the framework holds, it predicts that next-generation long-horizon, multi-agent, multimodal corpora will push the field toward memory-augmented and state-space models plus retrieval-augmented video understanding, since those structures directly target temporal span and compositionality.
  • Editorial inference: The authors' H/M/L ratings are assigned by hand; a community-validated or automatically computed scoring of dataset attributes would let the framework serve as a reusable diagnostic tool for predicting which architecture family suits any new benchmark.
  • Editorial inference: The same lens could be applied prospectively during dataset construction: deliberately vary the four attributes to probe whether an architecture's inductive bias is genuinely being challenged, rather than relying on leaderboard rankings alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This survey argues that the evolution of video understanding architectures is fundamentally shaped by dataset structure. It introduces a 'dataset-bias-architecture' framework in which four dataset properties—motion amplitude, temporal span, compositionality/hierarchy, and multimodal richness—impose inductive biases that drive architectural choices. The paper organizes major datasets into these categories (Table II), reviews milestones from two-stream CNNs and 3D CNNs through transformers and video-language models, and uses curated benchmark tables (Tables III and IV) to claim that dataset properties predict which model families succeed. It concludes with a prescriptive roadmap for aligning model design with dataset structure and for constructing future datasets.

Significance. If the central causal claim—that datasets are 'the principal structural force shaping model design'—were established, the survey would offer a useful organizing perspective for a fragmented literature. The paper has clear strengths: Table II is a broad compendium of datasets with structural annotations, the coverage of modern video-language and egocentric datasets is current, the authors provide code and dynamic visualizations, and the roadmap in Section V is actionable. However, the significance is currently undermined by the fact that the empirical support consists of selectively curated benchmark numbers and hand-assigned dataset ratings, with no controlled comparison. The central claim is plausible as a retrospective narrative but is not demonstrated at the strength asserted.

major comments (4)
  1. [Section IV.A, Table III] The text states that 'On HMDB51 and UCF101, early Two-Stream variants and 3D CNNs consistently outperform others,' citing Two-Stream'16 (69.2/93.5) and RGB-I3D (74.8/95.6). The same Table III lists VideoMAE V2 at 88.1 and 99.6 on these datasets, and InternVideo at 89.3 on HMDB51. These are 15–20 points higher, so the sentence is factually contradicted by the paper's own table. The table caption also says 'the best-performing model variant is reported,' meaning rows are not comparable under a fixed protocol: models differ in pretraining data, compute, input sampling, and evaluation settings. This contradiction undermines the inference that short-clip datasets 'strongly favor' two-stream/3D CNNs. A similar issue appears in Section IV.B, where early backbones are said to 'consistently excel' in temporal localization, while Table IV lists InternVideo2 at 72.0 mAP on THUMOS'14 versus I3D+Flow
  2. [Sections III.A and V.B] The central thesis that datasets are 'the principal structural force shaping model design' is asserted on the basis of a correlation between hand-assigned dataset attributes (Table II) and selected architecture successes (Tables III–IV). No attempt is made to hold constant model scale, pretraining data, compute budget, or evaluation protocol, so the observed alignment is equally consistent with compute-driven or pretraining-driven evolution. Moreover, Section V.A itself acknowledges that 'evaluation fragmentation' and leaderboard incentives 'shape architectural incentives'—a non-dataset confound. The causal claim is therefore not supported by the presented evidence. Please either weaken the claim to 'an important and underexamined influence' or provide a more rigorous argument, e.g., a historical timeline showing architecture transitions following dataset releases, or citations to ablati
  3. [Table II] The structural ratings (Amp/Span/Comp/Agents) are central to the framework, but they are assigned without an explicit rubric, operational definitions, inter-annotator agreement, or sensitivity analysis. For example, Kinetics-400 is rated Amp=H, Span=S, Comp=-, Agents=M, but the criteria for these levels are not given, and a different researcher could plausibly rate the same dataset differently. Because these ratings are used to support the paper's main narrative, their subjectivity is load-bearing. Please provide a coding protocol, report reliability, or explicitly relabel the ratings as informal and reduce their role in the causal argument.
  4. [Tables III and IV] The selective reporting in Tables III and IV makes it difficult to interpret 'dashes' as capabilities. For example, VideoMAE V2 has no retrieval or QA entries in Table IV, and InternVideo2 lacks QA entries, yet the text interprets such absences as evidence that certain model families are specialized or limited. A model with no reported number may simply have not been evaluated on that benchmark. The tables should include a completeness statement or a reference to the original papers' evaluation suites, and the text should avoid reading missing entries as negative evidence.
minor comments (5)
  1. [Section IV.A and References] The model 'Two-Stream'16' is cited as [184] (Feichtenhofer et al., 2016), but the original two-stream architecture is [38] (Simonyan and Zisserman, 2014). Please clarify the naming to avoid confusion between the two papers.
  2. [Table III] The row labeled 'Swin' refers to Video Swin Transformer [239]; using the unqualified name may be confused with the image Swin Transformer. Please rename to 'Video Swin'.
  3. [Table II] The UCF101-24 dataset is listed with year 2024, but UCF101-24 is a subset of UCF101 with spatio-temporal annotations and is much older. Please correct the year or clarify the provenance.
  4. [Front matter] The arXiv abstract uses the title 'Video Understanding by Design: How Datasets Shape Video Models,' while the manuscript header title is '... How Datasets Shape Architectures and Insights.' Please unify the title and abstract wording.
  5. [Figure 5] The caption says 'Images adopted from [180]'; for a journal submission please confirm that permission or license for reuse is obtained and that the source is clearly credited.

Circularity Check

0 steps flagged

No circularity: the survey's dataset-centric synthesis is an independent reading of external benchmark results, not a derivation from its own inputs.

full rationale

This is a survey with no equations, fitted parameters, or uniqueness theorems, so the classic failure modes (self-definitional equations, fitted inputs renamed as predictions, ansatz smuggled via citation) do not apply. The central claim—that dataset structure (motion complexity, temporal span, compositionality, multimodal richness) shaped architecture evolution—is supported by Table II's historical categorization and Tables III–IV, which compile externally published benchmark numbers. Those tables are not derived from the framework; they are independent evidence. The authors' self-citations (e.g., refs [4]–[13]) are contextual and not load-bearing for the dataset-centric thesis. The paper even acknowledges in Section V.A that evaluation fragmentation and leaderboard tuning shape architectural incentives, which weakens the causal exclusivity of the central claim but is a correctness/evidential concern, not circularity. Table III's 'best-performing model variant is reported' protocol may bias the qualitative reading, but selecting published results is not the same as fitting a parameter to the data being predicted. No specific reduction of a conclusion to an input by construction can be exhibited, so no circular step is identified.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The paper introduces a conceptual entity, 'structural pressure', but it is a framing device rather than a physical or mathematical entity with independent falsifiable evidence. The load-bearing assumptions are the causal arrow from datasets to architectures and the validity of the hand-coded dataset attribute ratings and compiled benchmark numbers.

free parameters (1)
  • Hand-assigned dataset attribute ratings (motion amplitude, temporal span, compositionality, agent density) in Table II = H/M/L per dataset, assigned by authors
    These categorical ratings are chosen without an explicit measurement protocol and are used to support the dataset-driven narrative. They are not fitted numbers but are hand-set values that the framework depends on.
axioms (3)
  • domain assumption The four structural pressures (motion complexity, temporal span, hierarchical structure, multimodal richness) are the dominant forces driving video model evolution, and they exclude class distribution, labeling schemes, and collection biases (footnote 1, Section I).
    This definition is the foundation of the framework and is asserted without empirical validation. The choice of which properties count as 'structural' is arbitrary and excludes alternative causes.
  • domain assumption Datasets induce inductive biases in architectures (Section III.A), meaning the causal arrow runs from data to model design.
    The survey assumes causality from dataset structure to architectural innovation. Alternative explanations such as compute growth, benchmark leaderboards, and theoretical insights are not controlled or refuted.
  • domain assumption Benchmark numbers compiled from different papers are accurate and mutually comparable (Tables III and IV).
    The empirical support relies on reported numbers from heterogeneous evaluation protocols. The paper does not standardize splits, input sizes, or pretraining, and it does not verify the numbers.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Video Understanding by Design: How Datasets Shape Video Models." pith.science (2026). https://pith.science/paper/FKHX2A4F

@misc{pith2026250909151,
  author       = {Pith},
  title        = {Pith review of: Video Understanding by Design: How Datasets Shape Video Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKHX2A4F}},
  note         = {Machine review of arXiv:2509.09151}
}
Share X LinkedIn Reddit HN
read the original abstract

Research in video understanding has advanced rapidly, driven by increasingly diverse datasets and more powerful model architectures. While existing surveys typically organize progress by tasks, benchmarks, or model families, they provide limited insight into why particular architectures emerged and succeeded. In this survey, we argue that the evolution of video understanding is fundamentally shaped by dataset structure. We present a dataset-centric perspective that connects dataset structure, inductive biases, and architectural design within a unified framework. We show that different datasets require models to capture specific invariances and capabilities, such as robustness to viewpoint changes, sensitivity to temporal ordering, reasoning over long-range dependencies, relational interactions, and cross-modal alignment. These requirements naturally give rise to inductive biases, i.e., architectural assumptions that favor particular patterns of reasoning and generalization. From this perspective, milestone architectures, including two-stream networks, 3D CNNs, temporal models, transformers, graph-based methods, and multimodal foundation models, can be understood as architectural responses to the challenges posed by evolving datasets. Building on this framework, we systematically analyze how dataset characteristics have shaped architectural innovation across video understanding tasks and discuss the representational biases induced by different data regimes. By unifying datasets, inductive biases, and architectures into a coherent perspective, this survey offers both a retrospective explanation of the field's evolution and a forward-looking roadmap toward general-purpose video understanding systems. Code and dynamic video visualizations of dataset-induced biases are available at https://time.griffith.edu.au/paper-sites/video-understanding/.

Figures

Figures reproduced from arXiv: 2509.09151 by Lei Wang, Piotr Koniusz, Syuan-Hao Li, Yongsheng Gao.

Figure 1
Figure 1. Figure 1: Datasets as structural lenses. Key attributes, motion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motion complexity across datasets. UCF101 (top) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Hierarchical and compositional structures in video datasets. (a) Kinetics-400: [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples of egocentric and long-horizon video datasets highlighting temporal and procedural complexity. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Koala-36M illustrates the power of large-scale, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

250 extracted references · 33 linked inside Pith

  1. [1]

    Large-scale video classification with convolutional neural networks,

    A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” inProceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014, pp. 1725–1732

    2014

  2. [2]

    Learning spatiotemporal features with 3d convolutional networks,

    D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” inProceedings of the IEEE international conference on computer vision, 2015, pp. 4489–4497

    2015

  3. [3]

    Slowfast networks for video recognition,

    C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6202–6211

    2019

  4. [4]

    Motion meets attention: Video motion prompts,

    Q. Chen, L. Wang, P. Koniusz, and T. Gedeon, “Motion meets attention: Video motion prompts,” inAsian Conference on Machine Learning. PMLR, 2025, pp. 591–606. 15

    2025

  5. [5]

    Taylor videos for action recognition,

    L. Wang, X. Yuan, T. Gedeon, and L. Zheng, “Taylor videos for action recognition,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 52 117–52 133

    2024

  6. [6]

    Learnable expansion of graph operators for multi-modal feature fusion,

    D. Ding, L. Wang, L. Zhu, T. Gedeon, and P. Koniusz, “Learnable expansion of graph operators for multi-modal feature fusion,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=SMZqIOSdlN

    2025

  7. [7]

    Meet jeanie: a similarity measure for 3d skeleton sequences via temporal-viewpoint alignment,

    L. Wang, J. Liu, L. Zheng, T. Gedeon, and P. Koniusz, “Meet jeanie: a similarity measure for 3d skeleton sequences via temporal-viewpoint alignment,”International Journal of Computer Vision, vol. 132, no. 9, pp. 4091–4122, 2024

    2024

  8. [8]

    Evolving skeletons: Motion dynamics in action recognition,

    J. Qiu and L. Wang, “Evolving skeletons: Motion dynamics in action recognition,” inCompanion Proceedings of the ACM on Web Confer- ence 2025, 2025, pp. 1916–1937

    2025

  9. [9]

    Feature hallucination for self-supervised action recognition,

    L. Wang and P. Koniusz, “Feature hallucination for self-supervised action recognition,”International Journal of Computer Vision, 2025

    2025

  10. [10]

    Do language models understand time?

    X. Ding and L. Wang, “Do language models understand time?” in Companion Proceedings of the ACM on Web Conference 2025, 2025, pp. 1855–1868

    2025

  11. [11]

    Quo vadis, anomaly detection? llms and vlms in the spotlight,

    ——, “Quo vadis, anomaly detection? llms and vlms in the spotlight,” arXiv preprint arXiv:2412.18298, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    The journey of action recognition,

    ——, “The journey of action recognition,” inCompanion Proceedings of the ACM on Web Conference 2025, 2025, pp. 1869–1884

    2025

  13. [13]

    Representation-centric survey of skeletal action recognition and the anubis benchmark,

    Y. Liu, J. Yang, M. Perera, P. Ji, D. Kim, M. Xu, T. Wang, S. Anwar, T. Gedeon, L. Wanget al., “Representation-centric survey of skeletal action recognition and the anubis benchmark,”CoRR, 2025

    2025

  14. [14]

    Foundation models for video understanding: A survey,

    N. Madan, A. Møgelmose, R. Modi, Y. S. Rawat, and T. B. Moeslund, “Foundation models for video understanding: A survey,”arXiv preprint arXiv:2405.03770, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Video understanding with large language models: A survey,

    Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhuet al., “Video understanding with large language models: A survey,”IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  16. [16]

    The kinetics human action video dataset,

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijaya- narasimhan, F. Viola, T. Green, T. Back, P. Natsevet al., “The kinetics human action video dataset,”arXiv preprint arXiv:1705.06950, 2017

    Pith/arXiv arXiv 2017

  17. [17]

    The” something something

    R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitaget al., “The” something something” video database for learning and evaluating visual common sense,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 5842–5850

    2017

  18. [18]

    Activi- tynet: A large-scale video benchmark for human activity understanding,

    F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “Activi- tynet: A large-scale video benchmark for human activity understanding,” inProceedings of the ieee conference on computer vision and pattern recognition, 2015, pp. 961–970

    2015

  19. [19]

    Hollywood in homes: Crowdsourcing data collection for activity understanding,

    G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” inEuropean conference on computer vision. Springer, 2016, pp. 510–526

    2016

  20. [20]

    Charades-ego: A large-scale dataset of paired third and first person videos,

    G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari, “Charades-ego: A large-scale dataset of paired third and first person videos,”arXiv preprint arXiv:1804.09626, 2018

    Pith/arXiv arXiv 2018

  21. [21]

    Ava: A video dataset of spatio-temporally localized atomic visual actions,

    C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankaret al., “Ava: A video dataset of spatio-temporally localized atomic visual actions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6047–6056

    2018

  22. [22]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100,

    D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Priceet al., “Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100,”International Journal of Computer Vision, vol. 130, no. 1, pp. 33–55, 2022

    2022

  23. [23]

    Graph based skeleton motion representation and similarity measurement for action recognition,

    P. Wang, C. Yuan, W. Hu, B. Li, and Y. Zhang, “Graph based skeleton motion representation and similarity measurement for action recognition,” inEuropean conference on computer vision. Springer, 2016, pp. 370–385

    2016

  24. [24]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308

    2017

  25. [25]

    Non-local neural networks,

    X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7794–7803

    2018

  26. [26]

    Spatial temporal graph convolutional networks for skeleton-based action recognition,

    S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

    2018

  27. [27]

    Vivit: A video vision transformer,

    A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lu ˇci´c, and C. Schmid, “Vivit: A video vision transformer,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6836–6846

    2021

  28. [28]

    Is space-time attention all you need for video understanding?

    G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” inIcml, vol. 2, no. 3, 2021, p. 4

    2021

  29. [29]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training,

    Z. Tong, Y. Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training,”Advances in neural information processing systems, vol. 35, pp. 10 078–10 093, 2022

    2022

  30. [30]

    Internvideo: General video foundation models via gen- erative and discriminative learning,

    Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wanget al., “Internvideo: General video foundation models via gen- erative and discriminative learning,”arXiv preprint arXiv:2212.03191, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Videomae v2: Scaling video masked autoencoders with dual masking,

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao, “Videomae v2: Scaling video masked autoencoders with dual masking,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14 549–14 560

    2023

  32. [32]

    Internvideo2: Scaling foundation models for multimodal video understanding,

    Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng, Z. Wang, Y. Shiet al., “Internvideo2: Scaling foundation models for multimodal video understanding,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 396–416

    2024

  33. [33]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,”arXiv preprint arXiv:1212.0402, 2012

    Pith/arXiv arXiv 2012

  34. [34]

    Hmdb: a large video database for human motion recognition,

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: a large video database for human motion recognition,” in2011 International conference on computer vision. IEEE, 2011, pp. 2556–2563

    2011

  35. [35]

    A spatio-temporal descriptor based on 3d-gradients,

    A. Klaser, M. Marsza lek, and C. Schmid, “A spatio-temporal descriptor based on 3d-gradients,” inBMVC 2008-19th British machine vision conference. British Machine Vision Association, 2008, pp. 275–1

    2008

  36. [36]

    Action recognition with improved trajectories,

    H. Wang and C. Schmid, “Action recognition with improved trajectories,” inProceedings of the IEEE international conference on computer vision, 2013, pp. 3551–3558

    2013

  37. [37]

    Scaling egocentric vision: The epic-kitchens dataset,

    D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Priceet al., “Scaling egocentric vision: The epic-kitchens dataset,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 720–736

    2018

  38. [38]

    Two-stream convolutional networks for action recognition in videos,

    K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,”Advances in neural information processing systems, vol. 27, 2014

    2014

  39. [39]

    Omnivl: One foundation model for image-language and video-language tasks,

    J. Wang, D. Chen, Z. Wu, C. Luo, L. Zhou, Y. Zhao, Y. Xie, C. Liu, Y.-G. Jiang, and L. Yuan, “Omnivl: One foundation model for image-language and video-language tasks,”Advances in neural information processing systems, vol. 35, pp. 5696–5710, 2022

    2022

  40. [40]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks,

    J. Wu, M. Zhong, S. Xing, Z. Lai, Z. Liu, Z. Chen, W. Wang, X. Zhu, L. Lu, T. Luet al., “Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks,”Advances in Neural Information Processing Systems, vol. 37, pp. 69 925–69 975, 2024

    2024

  41. [41]

    Timechat: A time-sensitive multimodal large language model for long video understanding,

    S. Ren, L. Yao, S. Li, X. Sun, and L. Hou, “Timechat: A time-sensitive multimodal large language model for long video understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 313–14 323

    2024

  42. [42]

    Videollama 3: Frontier multimodal foundation models for image and video understanding,

    B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Liet al., “Videollama 3: Frontier multimodal foundation models for image and video understanding,”arXiv preprint arXiv:2501.13106, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action,

    J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi, “Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 439–26 455

    2024

  44. [44]

    Masked feature prediction for self-supervised visual pre-training,

    C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 14 668–14 678

    2022

  45. [45]

    Transductive zero-shot action recog- nition by word-vector embedding,

    X. Xu, T. Hospedales, and S. Gong, “Transductive zero-shot action recog- nition by word-vector embedding,”International Journal of Computer Vision, vol. 123, no. 3, pp. 309–333, 2017

    2017

  46. [46]

    Out-of-distribution detection for generalized zero-shot action recognition,

    D. Mandal, S. Narayan, S. K. Dwivedi, V. Gupta, S. Ahmed, F. S. Khan, and L. Shao, “Out-of-distribution detection for generalized zero-shot action recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9985–9993

    2019

  47. [47]

    Few-shot action recognition with permutation-invariant attention,

    H. Zhang, L. Zhang, X. Qi, H. Li, P. H. Torr, and P. Koniusz, “Few-shot action recognition with permutation-invariant attention,” inEuropean conference on computer vision. Springer, 2020, pp. 525–542. 16

    2020

  48. [48]

    Actionclip: A new paradigm for video action recognition,

    M. Wang, J. Xing, and Y. Liu, “Actionclip: A new paradigm for video action recognition,”arXiv preprint arXiv:2109.08472, 2021

    Pith/arXiv arXiv 2021

  49. [49]

    Temporal-relational crosstransformers for few-shot action recognition,

    T. Perrett, A. Masullo, T. Burghardt, M. Mirmehdi, and D. Damen, “Temporal-relational crosstransformers for few-shot action recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 475–484

    2021

  50. [50]

    Temporal-viewpoint transportation plan for skeletal few-shot action recognition,

    L. Wang and P. Koniusz, “Temporal-viewpoint transportation plan for skeletal few-shot action recognition,” inProceedings of the Asian conference on computer vision, 2022, pp. 4176–4193

    2022

  51. [51]

    Uncertainty-dtw for time series and sequences,

    ——, “Uncertainty-dtw for time series and sequences,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 176–195

    2022

  52. [52]

    Reinforced video captioning with entail- ment rewards,

    R. Pasunuru and M. Bansal, “Reinforced video captioning with entail- ment rewards,”arXiv preprint arXiv:1708.02300, 2017

    Pith/arXiv arXiv 2017

  53. [53]

    Embodied question answering,

    A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, “Embodied question answering,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1–10

    2018

  54. [54]

    Merlot: Multimodal neural script knowledge models,

    R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi, “Merlot: Multimodal neural script knowledge models,” Advances in neural information processing systems, vol. 34, pp. 23 634– 23 651, 2021

    2021

  55. [55]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  56. [56]

    Human action recognition from various data modalities: A review,

    Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 3, pp. 3200–3225, 2022

    2022

  57. [57]

    Video-language understanding: A survey from model architecture, model training, and data perspectives,

    T. Nguyen, Y. Bin, J. Xiao, L. Qu, Y. Li, J. Z. Wu, C.-D. Nguyen, S.-K. Ng, and L. A. Tuan, “Video-language understanding: A survey from model architecture, model training, and data perspectives,”arXiv preprint arXiv:2406.05615, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Video question answering: A survey of the state-of-the-art,

    J. P.J. and B. C. Kovoor, “Video question answering: A survey of the state-of-the-art,”J. Vis. Comun. Image Represent., vol. 105, no. C, Dec

  59. [59]

    A survey on generative ai and llm for video generation, understanding, and streaming,

    P. Zhou, L. Wang, Z. Liu, Y. Hao, P. Hui, S. Tarkoma, and J. Kangasharju, “A survey on generative ai and llm for video generation, understanding, and streaming,”arXiv preprint arXiv:2404.16038, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Human activity analysis: A review,

    J. K. Aggarwal and M. S. Ryoo, “Human activity analysis: A review,” Acm Computing Surveys (Csur), vol. 43, no. 3, pp. 1–43, 2011

    2011

  61. [61]

    Going deeper into action recognition: A survey,

    S. Herath, M. Harandi, and F. Porikli, “Going deeper into action recognition: A survey,”Image and vision computing, vol. 60, pp. 4– 21, 2017

    2017

  62. [62]

    A survey on video-based human action recognition: recent updates, datasets, challenges, and applications,

    P. Pareek and A. Thakkar, “A survey on video-based human action recognition: recent updates, datasets, challenges, and applications,” Artificial Intelligence Review, vol. 54, no. 3, pp. 2259–2322, 2021

    2021

  63. [63]

    Vision transformers for action recognition: A survey,

    A. Ulhaq, N. Akhtar, G. Pogrebna, and A. Mian, “Vision transformers for action recognition: A survey,”arXiv preprint arXiv:2209.05700, 2022

    Pith/arXiv arXiv 2022

  64. [64]

    Video transformers: A survey,

    J. Selva, A. S. Johansen, S. Escalera, K. Nasrollahi, T. B. Moeslund, and A. Clap ´es, “Video transformers: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 12 922– 12 943, 2023

    2023

  65. [65]

    End-to-end learning of visual representations from uncurated instruc- tional videos,

    A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman, “End-to-end learning of visual representations from uncurated instruc- tional videos,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9879–9889

    2020

  66. [66]

    Multimodal learning with transform- ers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transform- ers: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 113–12 132, 2023

    2023

  67. [67]

    A survey on human activity recognition from videos,

    T. Subetha and S. Chitrakala, “A survey on human activity recognition from videos,” in2016 international conference on information commu- nication and embedded systems (ICICES). IEEE, 2016, pp. 1–7

    2016

  68. [68]

    A comparative review of recent kinect-based action recognition algorithms,

    L. Wang, D. Q. Huynh, and P. Koniusz, “A comparative review of recent kinect-based action recognition algorithms,”IEEE Transactions on Image Processing, vol. 29, pp. 15–28, 2019

    2019

  69. [69]

    Skeleton- based action recognition with shift graph convolutional network,

    K. Cheng, Y. Zhang, X. He, W. Chen, J. Cheng, and H. Lu, “Skeleton- based action recognition with shift graph convolutional network,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 183–192

    2020

  70. [70]

    Graph convo- lutional neural network for human action recognition: A comprehensive survey,

    T. Ahmad, L. Jin, X. Zhang, S. Lai, G. Tang, and L. Lin, “Graph convo- lutional neural network for human action recognition: A comprehensive survey,”IEEE Transactions on Artificial Intelligence, vol. 2, no. 2, pp. 128–145, 2021

    2021

  71. [71]

    A survey on deep learning for skeleton-based human animation,

    L. Mourot, L. Hoyet, F. Le Clerc, F. Schnitzler, and P. Hellier, “A survey on deep learning for skeleton-based human animation,” inComputer Graphics Forum, vol. 41, no. 1. Wiley Online Library, 2022, pp. 122– 157

    2022

  72. [72]

    A survey on 3d skeleton-based action recognition using learning method,

    B. Ren, M. Liu, R. Ding, and H. Liu, “A survey on 3d skeleton-based action recognition using learning method,”Cyborg and Bionic Systems, vol. 5, p. 0100, 2024

    2024

  73. [73]

    Self-supervised visual feature learning with deep neural networks: A survey,

    L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 11, pp. 4037–4058, 2020

    2020

  74. [74]

    Self- supervised learning: Generative or contrastive,

    X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self- supervised learning: Generative or contrastive,”IEEE transactions on knowledge and data engineering, vol. 35, no. 1, pp. 857–876, 2021

    2021

  75. [75]

    Self-supervised representation learning: Introduction, advances, and challenges,

    L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, “Self-supervised representation learning: Introduction, advances, and challenges,”IEEE Signal Processing Magazine, vol. 39, no. 3, pp. 42–62, 2022

    2022

  76. [76]

    Deep generative models: Survey,

    A. Oussidi and A. Elhassouny, “Deep generative models: Survey,” in2018 International conference on intelligent systems and computer vision (ISCV). IEEE, 2018, pp. 1–8

    2018

  77. [77]

    A survey of multimodal deep generative models,

    M. Suzuki and Y. Matsuo, “A survey of multimodal deep generative models,”Advanced Robotics, vol. 36, no. 5-6, pp. 261–278, 2022

    2022

  78. [78]

    Sora as an agi world model? a complete survey on text-to-video generation,

    J. Cho, F. D. Puspitasari, S. Zheng, J. Zheng, L.-H. Lee, T.-H. Kim, C. S. Hong, and C. Zhang, “Sora as an agi world model? a complete survey on text-to-video generation,”arXiv preprint arXiv:2403.05131, 2024

    arXiv 2024

  79. [79]

    A survey on video diffusion models,

    Z. Xing, Q. Feng, H. Chen, Q. Dai, H. Hu, H. Xu, Z. Wu, and Y.-G. Jiang, “A survey on video diffusion models,”ACM Computing Surveys, vol. 57, no. 2, pp. 1–42, 2024

    2024

  80. [80]

    Benchmarking a multimodal and multiview and interactive dataset for human action recognition,

    A.-A. Liu, N. Xu, W.-Z. Nie, Y.-T. Su, Y. Wong, and M. Kankanhalli, “Benchmarking a multimodal and multiview and interactive dataset for human action recognition,”IEEE Transactions on cybernetics, vol. 47, no. 7, pp. 1781–1794, 2016

    2016

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.