Pith. sign in

REVIEW 4 major objections 6 minor 72 references

The paper claims that LLM-generated atomic actions — fine-grained spatial-temporal descriptions of each action label — injected into scene-graph video features improve multi-label action recognition beyond prompt-only disentanglement.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:56 UTC pith:QSF32PLD

load-bearing objection The oracle numbers rest on an undefined test-time Action Specification, and the standard Charades row trails ProDA; the idea is worth engaging, but the SOTA claim doesn't hold as written. the 4 major comments →

arxiv 2607.26097 v1 pith:QSF32PLD submitted 2026-07-28 cs.CV

Knowledge-guided Disentanglement with Atomic Actions for Action Recognition

Tianci Wu , Siqi Cao , Guangming Zhu , Jiang Lu , Siyuan Wang , Longfei Zhang , Jincai Huang , Jun Sheng
show 1 more author
Liang Zhang
This is my paper
classification cs.CV
keywords action recognitionmulti-label classificationdisentangled representation learningatomic actionsknowledge injectionscene graphslarge language modelsvideo understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that prompt-based action disentanglement falls short because it lacks explicit semantic guidance, and that LLM-generated 'atomic actions' can supply that guidance. It proposes KDA, which decomposes each action label into fine-grained spatial-temporal descriptions, stores them in memory banks, adaptively retrieves and injects them into scene-graph node features (KIM), and re-retrieves them under specified/unspecified action prompts through a disentanglement module (KDM) regularized by a knowledge-disentanglement loss. The central claim is that this hierarchical knowledge guidance produces more discriminative, better-separated action features — reported as 73.2 mAP on Charades with oracle scene graphs versus 71.1 for the prior prompt-only method, and 11.70 mAP on SportsHHI versus a 10.69 baseline. A sympathetic reader would care because the two modules are presented as plug-in additions to existing models, meaning fine-grained semantic knowledge could improve action recognition broadly.

Core claim

On its own terms, the paper finds that injecting fine-grained semantic knowledge into scene graphs, then re-disentangling it under action-specification prompts, yields more discriminative features for multi-label action recognition. The central discovery is a balance: atomic actions alone inject overly strong semantics that overwhelm visual features (69.35 mAP), label-level knowledge is too coarse (70.96), and only atomic actions combined with learnable embeddings reach the best result (73.2 mAP under oracle scene graphs vs 71.1 for the prior prompt-only method). A margin-bounded knowledge-disentanglement loss over parallel heads keeps components mutually independent yet aligned to semantic

What carries the argument

The load-bearing object is the Atomic Action Memory Bank: each action label is decomposed by an LLM into temporally ordered spatial-temporal descriptions, encoded by a text encoder, and stored per-step in memory banks. A scoring network with a straight-through estimator thresholds a variable number of atomic entries per scene-graph node, which is how overlapping actions share nodes without node-level supervision. In KDM the same selection is masked by the Action Specification (the multi-hot SAP/UAP pair), so only knowledge relevant to specified or residual actions is retrieved; learnable embeddings fused alongside buffer the semantics so they do not dominate. The Knowledge Disentanglement Lo

Load-bearing premise

The load-bearing premise is that the Action Specification prompts steering disentanglement can be built at test time without ground-truth labels: the supplementary material constructs SAP/UAP prompts from ground-truth label sets, and in the standard protocol, where scene graphs are predicted, KDA's 49.9 mAP trails the prompt-only baseline's 50.2 — so the reported headline gains rest on this unstated prompt-construction assumption.

What would settle it

A controlled run of KDA on Charades using oracle scene graphs but with SAP/UAP prompts built from the model's own predictions (or from a single fixed full-label prompt) rather than ground-truth labels: if mAP falls from 73.2 toward the standard-setting level around 50, the gains attributed to atomic-action knowledge are driven by label identity leaking through the prompts rather than by the semantic descriptions themselves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding KIM and KDM to existing action-recognition models improves mAP and recall (the paper reports gains on SlowFast and the SportsHHI baseline), so fine-grained knowledge injection transfers across backbones.
  • Atomic action guidance helps most for coverage-sensitive metrics like R@20/R@50 and for classes absent from training, so the benefit should grow when the label space is broad or open.
  • The gains do not depend on a specific LLM: comparable results across three LLMs and structured prompt templates show the knowledge source is interchangeable, while an unstructured template is consistently worse.
  • Even when scene-graph relationship annotations are removed, atomic actions supply enough relational information to stay near the full-graph performance (70.9 vs 73.2 mAP), suggesting the semantic descriptions partially replace structured annotations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the headline comparison is an oracle setting — both scene graphs and the Action Specification prompts are built from ground-truth labels; under the standard protocol KDA's 49.9 mAP sits below the prior prompt-only method's 50.2. A direct test is to build test-time SAP/UAP prompts from the model's own predictions or a fixed prompt and measure how much of the oracle gain survive
  • Editorial inference: the retrieve–inject–disentangle pattern could generalize to other tasks with text-decomposable factors (group activities, human-object interactions, assembly or surgical steps), provided the knowledge strength is calibrated so the semantics do not overwhelm the visual encoder.
  • Editorial inference: the reported head-routing statistics point to capacity allocation as the performance ceiling; a dynamic head-count or per-node head selection mechanism could avoid the under-capacity (8 heads) and over-capacity (14 heads) regimes the paper observes.
  • Editorial inference: the ablations show more semantic detail is not uniformly better — atomic actions alone hurt — so an explicit search over knowledge granularity (number of atomic steps, sentence vs. phrase level) would map the sweet spot the paper identifies only qualitatively.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes KDA, a framework for multi-label action recognition that uses LLM-generated 'atomic actions' to inject fine-grained semantic knowledge into spatial-temporal scene graph features (KIM) and to condition a prompt-based disentanglement module (KDM) via a Specified/Unspecified Action Prompt pair. A novel Knowledge Disentanglement Loss regularizes the disentanglement heads. Experiments on Charades and SportsHHI report gains over ProDA in the oracle Charades setting (73.2 vs 71.1 mAP) and on SportsHHI (11.70 vs 10.69 mAP), but the standard Charades result is 49.9 mAP versus ProDA's 50.2.

Significance. The core idea of using LLM-generated atomic actions to provide explicit semantic guidance for action disentanglement is timely and potentially useful; the hierarchical feature/structure/semantic design is a natural extension of ProDA. The paper includes several ablations, an efficiency analysis, and a parameter-matched comparison on SportsHHI, which are helpful. However, the paper's central claim of state-of-the-art performance is currently not supported by the non-oracle experiments, and a load-bearing protocol detail—how Action Specifications are constructed at test time—is left unspecified. If the authors can clarify and fix that protocol, the contribution may be valuable; in its present form the empirical evidence is inconclusive.

major comments (4)
  1. [§4.2, §A.1–A.2, Eq. (6), Eq. (10)] The test-time construction of the Action Specification is never defined. The supplementary builds SAPs from ground-truth label sets (Eqs. 16–25) and UAP as their complement, and KDM consumes this AS at inference through Eq. (6) and Eq. (10). If test-time AS is constructed the same way, the oracle Charades gain (73.2 vs ProDA 71.1) is confounded by label leakage: the model is handed the set of present actions before classification. The only non-leak comparison is the standard Charades row in Table 1, where KDA (49.9) trails ProDA (50.2). Please specify how p_sap and p_uap are obtained at test time without ground-truth labels, and report results under that protocol.
  2. [Abstract, §1, Table 1] The abstract and contributions claim state-of-the-art performance on multi-label action recognition benchmarks. Table 1 shows KDA under the standard protocol at 49.9 mAP, below ProDA at 50.2, and the text itself describes this as 'comparable.' The SOTA claim is therefore supported only by the oracle row, which uses ground-truth scene graphs and, per the supplementary, ground-truth labels in the AS. This overstates the evidence. Revise the claim or provide a standard-protocol result with a positive margin; additionally, note that the oracle and standard rows use different visual features (CLIP vs LaIAR), further complicating cross-setting inference.
  3. [§3.2–§3.4, Tables 1–2] Reproducibility and uncertainty: the main results are single runs without error bars, and the key differences are small relative to typical run-to-run variance (49.9 vs 50.2 on Charades standard; +1.01 mAP on SportsHHI). Hyperparameters introduced in Section 3—τ, M, L, t, γ, λ1, λ2, m1, m2—are not reported; only M and L are ablated in the supplementary. The paper states 'The code is available here' but no URL is given. These omissions prevent independent verification of the central empirical claim.
  4. [§2.2, §3.1, Table 4] The atomic actions are generated from the supervised action labels via an LLM, so the injected 'knowledge' is semantically derived from the prediction target itself. This does not invalidate the approach, but the paper should frame the contribution as a richer conditioning signal rather than independent external knowledge. Table 4 compares label versus atomic-action knowledge in the oracle setting; a standard-setting version of this comparison and a parameter-matched label-only baseline would strengthen the claim that gains come from atomic-action granularity rather than model capacity.
minor comments (6)
  1. [Figure 1] The caption lists parts (a) and (b), but the text in §1 and §3.1 refers to 'Fig. 1(c)' twice. Either add panel (c) to the figure/caption or fix the references.
  2. [Eq. (14) vs Supplementary Eq. (29)] The KD Loss in the supplementary (Eq. 29) writes h_m^* in the denominator while the main text (Eq. 14) uses h_m. Clarify whether these are the same quantity or whether normalization is intended; as written, the two equations are inconsistent.
  3. [Table 1] The caption defines 'SSG: ground-truth SSG' which is tautological; presumably SSG stands for 'scene graph.' Also define 'Bbox' and clarify which rows use predicted versus ground-truth scene graphs.
  4. [Table 2] Rows labeled 'SportsHHI (V)[53]' and 'SportsHHI (V)[52]' are confusing: [53] is the ProDA reference, not a SportsHHI baseline variant. Clearly identify the backbone and source for each row.
  5. [Author affiliations and code link] Typos: 'Human Institute of Advanced Technologyy' and lowercase 'Tianci wu' should be fixed. The phrase 'The code is available here' should include a working URL.
  6. [Table 5] The text claims that the method 'attains the highest minimum accuracy,' but the table reports only mean accuracy and standard deviation. Add a minimum-accuracy column or remove the claim.

Circularity Check

1 steps flagged

SOTA claim rests on unstated test-time Action Specification: §A builds SAP/UAP from ground-truth labels, and standard Charades (49.9) trails ProDA (50.2).

specific steps
  1. other [Supplementary §A.1–A.2; §3.3 Eq. (6), Eq. (10); Table 1]
    "Given a video with ground-truth label set{0, 1}, its multi-hot representation is [1, 1, 0, 0, 0]. ... Based on the ground-truth labels, we first enumerate all non-empty subsets of the label set. ... Given a video with ground-truth label set {77, 79, 80, 76, 75}, present labels are sampled from this set, while absent labels are sampled from the remaining label space... To prevent information leakage, the corresponding UAP is defined as the complement of each SAP in the full label space."

    The only Action Specification protocol given in the paper is built from ground-truth labels: SAP is a multi-hot vector of sampled true labels and UAP is its complement. KDM then conditions everything on this AS via Eq. 6 (˜δ = δ ⊙ p_as) and Eq. 10 (ˆf_j_{i,as} = MLP([f_j_i; g_as])). If this same construction is used at oracle evaluation — and the paper never specifies any other test-time AS protocol — the model is handed the true label set as an input before classifying those labels. The oracle Charades gain over ProDA (73.2 vs 71.1) is therefore at least partly attributable to label leakage rather than to atomic-action disentanglement. The non-oracle Charades row (49.9) is below ProDA (50.2), so the abstract's 'state-of-the-art' claim rests on this unspecified, leak-prone setting.

full rationale

The paper's derivation chain does not contain a fitted parameter that is renamed as a prediction, and there is no imported uniqueness theorem from the authors' prior work. The atomic actions are generated from action labels, but using label semantics as text input is a standard class-name-conditioning strategy and is not itself circular. The self-citations to ProDA [53] are architectural (VGPNN, AD Loss, random-sampling AS) rather than load-bearing external evidence; the central comparison is on held-out benchmark splits. The main circularity-adjacent issue is the Action Specification protocol: the supplement builds SAP/UAP exclusively from ground-truth label sets, while the main text never states how AS is produced at test time. Since Eq. 6 and Eq. 10 feed AS directly into the disentanglement and guided feature computation, an oracle evaluation that reuses the §A construction would put the prediction target into the input. This is a missing protocol step with direct potential for label leakage, not a fully demonstrated reduction by construction; the standard Charades result also trails ProDA, so the SOTA claim is fragile. Weighing these, the paper is only partially circular: score 3.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 2 invented entities

The framework assumes LLM-generated text knowledge is semantically aligned with visual SSG features, that STE thresholding preserves useful gradients, and that the SAP/UAP decomposition can be applied at test time. Several hyperparameters are tuned per dataset; the semantic anchors and memory banks are learned/invented structures without external independent evidence.

free parameters (7)
  • threshold tau = not reported
    Hard-threshold in KIM/KDM selection (Eq. 3-4); chosen by hand, no value given in main text.
  • AD margins m1, m2 = not reported
    Margins in Eq. 12 prevent over-penalization and representation collapse; values not stated.
  • loss weights lambda1, lambda2 = not reported
    Balance terms in Eq. 15; values not stated.
  • KD temperature t = not reported
    Temperature in KD loss Eq. 14; value not stated.
  • KD margin gamma = not reported
    Similarity clipping in Eq. 13; value not stated.
  • disentanglement heads M / semantic anchors L = M=10, L=64
    Selected by Charades ablation (Table 8); M=12 gives 73.25 vs 73.17 but M=10 chosen for efficiency.
  • LLM and prompt template = Qwen3-plus + ST prompt (main paper)
    Ablated in Table 13; the ST prompt is used in the paper, and the choice affects atomic-action quality.
axioms (6)
  • domain assumption LLM-generated atomic action descriptions from labels are semantically faithful and useful for visual action recognition.
    Core premise of KIM/KDM; §3.1 and Table 13 test prompt variants but not the faithfulness assumption itself.
  • domain assumption CLIP's text embedding space aligns atomic-action text with SSG node visual features.
    Knowledge retrieval scoring in §3.2 (Eq. 2) computes relevance between node features and CLIP text embeddings; no text-encoder fine-tuning is reported.
  • standard math Straight-through estimator gradient surrogate for discrete selection is adequate.
    STE [4] is used in Eq. 4 to backpropagate through thresholding; a known but heuristic technique.
  • domain assumption SAP/UAP decomposition with distractor labels prevents information leakage and supports multi-label disentanglement at test time.
    Supplementary §A constructs SAPs from ground-truth label sets; test-time applicability is not demonstrated.
  • domain assumption ProDA's VGPNN and AD loss (Pearson correlation + reconstruction) provide a valid disentanglement backbone.
    Adopted verbatim from self-cited [53]; Eq. 12 depends on this prior method's correctness.
  • domain assumption SSG construction (predicted or oracle) provides sufficient relational signal for atomic-knowledge injection.
    Standard protocol uses predicted SSGs from LaIAR features; oracle uses ground-truth SSGs; Table 6 shows 2.3 mAP drop without relationships.
invented entities (2)
  • Atomic Action Memory Banks (AMBs) no independent evidence
    purpose: Store CLIP embeddings of LLM-generated atomic action descriptions per class.
    They are derived offline from action labels via LLM and CLIP; no independent source outside the labels validates their semantics.
  • Learnable semantic anchors U no independent evidence
    purpose: Anchor shared information in the KD Loss while preserving inter-head diversity.
    Learnable prototypes trained on the dataset; no external handle or predicted observable.

pith-pipeline@v1.3.0-alltime-deepseek · 24311 in / 13706 out tokens · 129306 ms · 2026-08-01T02:56:38.546034+00:00 · methodology

0 comments
read the original abstract

Action recognition in complex scenes often involves multiple concurrent fine-grained actions, making it challenging to model internal action structures. Most existing methods rely on holistic representations, which are insufficient for capturing subtle interactions and fine-grained semantics. While recent prompt-based approaches introduce disentanglement, they lack explicit semantic guidance, and methods based solely on visual or structured cues remain coarse-grained. In this paper, we propose Knowledge-guided Disentanglement with Atomic Actions (KDA), which leverages fine-grained semantic knowledge to enhance action representations and enable more precise disentanglement. Specifically, we use Large Language Models (LLMs) to decompose action labels into atomic actions, providing explicit spatial-temporal semantics. A Knowledge Injection Module (KIM) first integrates atomic action knowledge into video features. Based on this enhanced representation, a Knowledge Disentanglement Module (KDM) further disentangles atomic action knowledge to produce more precise semantic guidance for action disentanglement. A Knowledge Disentanglement Loss (KD Loss) is introduced to encourage clearer disentanglement of knowledge components within KDM. Extensive experiments demonstrate that KDA improves feature discriminability and achieves state-of-the-art performance on multi-label action recognition benchmarks. Moreover, KIM and KDM can be readily integrated into other methods, demonstrating strong generality.

Figures

Figures reproduced from arXiv: 2607.26097 by Guangming Zhu, Jiang Lu, Jincai Huang, Jun Sheng, Liang Zhang, Longfei Zhang, Siqi Cao, Siyuan Wang, Tianci Wu.

Figure 1
Figure 1. Figure 1: (a) Traditional knowledge-guided paradigms based [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) An overview of our method. We first construct SSGs and extract node features using a video encoder. Meanwhile, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of disentangled knowledge in KDM [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An illustration of action specification (AS) construc [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of disentangled knowledge in KDM under KD Loss with and without a margin constraint, using 10 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 7 linked inside Pith

  1. [1]

    Amir Aghdam, Vincent Tao Hu, and Björn Ommer. 2025. Actalign: Zero-shot fine-grained video classification via language-guided sequence alignment. arXiv preprint arXiv:2506.22967 (2025)

    arXiv 2025

  2. [2]

    Mirza Mohtashim Alam, Md Rashad Al Hasan Rony, Mojtaba Nayyeri, Kar- ishma Mohiuddin, MST Mahfuja Akter, Sahar Vahdati, and Jens Lehmann. 2022. Language model guided knowledge graph embeddings. IEEE Access 10 (2022), 76008–76020

    2022

  3. [3]

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision . 6836–6846

    2021

  4. [4]

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

    Pith/arXiv arXiv 2013

  5. [5]

    Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6299–6308

    2017

  6. [6]

    Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Berta- sius. 2023. Vindlu: A recipe for effective video-and-language pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. 10739–10750

    2023

  7. [7]

    Y. Cong, W. Liao, H. Ackermann, et al. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 16372–16382

    2021

  8. [8]

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2023. Vision transformers need registers. arXiv preprint arXiv:2309.16588 (2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Emily L Denton et al. 2017. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems 30 (2017)

    2017

  10. [10]

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, et al. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6202–6211

    2019

  11. [11]

    Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. 2017. Spatiotemporal multiplier networks for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4768–4777

    2017

  12. [12]

    Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. 2016. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1933–1941

    2016

  13. [13]

    Lianqiang Gan, Junyu Lai, Jingze Ju, Lianli Gao, and Yi Bin. 2025. DFDNet: Disen- tangling and Filtering Dynamics for Enhanced Video Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 39. 3059–3067

    2025

  14. [14]

    Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. 2019. Video action transformer network. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 244–253

    2019

  15. [15]

    Arthur Gretton, Alexander Smola, Olivier Bousquet, Ralf Herbrich, Andrei Belit- ski, Mark Augath, Yusuke Murayama, Jon Pauls, Bernhard Schölkopf, and Nikos Logothetis. 2005. Kernel constrained covariance for dependence measurement. In International Workshop on Artificial Intelligence and Statistics . PMLR, 112–119

    2005

  16. [16]

    Mingfei Han, David Junhao Zhang, Yali Wang, Rui Yan, Lina Yao, Xiaojun Chang, and Yu Qiao. 2022. Dual-AI: Dual-path actor interaction learning for group activity recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2990–2999

    2022

  17. [17]

    Tal Hassner. 2013. A critical review of action recognition benchmarks. Inproceed- ings of the IEEE conference on computer vision and pattern recognition workshops . 245–250

    2013

  18. [18]

    Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles

  19. [19]

    Ji Ji, Rishabh Desai, and Juan Carlos Niebles. 2021. Detecting human-object relationships in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8106–8116

    2021

  20. [20]

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2020. Action genome: Actions as compositions of spatio-temporal scene graphs. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10236– 10247

    2020

  21. [21]

    Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan. 2020. Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5194–5202

    2020

  22. [22]

    Yang Jin, Linchao Zhu, and Yadong Mu. 2022. Complex video action reasoning via learnable markov logic network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3242–3251

    2022

  23. [23]

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022. Prompting visual-language models for efficient video understanding. In European conference on computer vision. Springer, 105–124

    2022

  24. [24]

    Kumara Kahatapitiya, Anurag Arnab, Arsha Nagrani, and Michael S Ryoo. 2024. Victr: Video-conditioned text representations for activity recognition. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18547–18558

    2024

  25. [25]

    Haoyang Li, Xin Wang, Zeyang Zhang, Haibo Chen, Ziwei Zhang, and Wenwu Zhu. 2024. Disentangled graph self-supervised learning for out-of-distribution generalization. In Forty-first International Conference on Machine Learning

    2024

  26. [26]

    Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu

  27. [27]

    Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. 2022. Disentangled graph contrastive learning with independence promotion. IEEE Transactions on Knowledge and Data Engineering 35, 8 (2022), 7856–7869

    2022

  28. [28]

    Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang

  29. [29]

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2022. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4804–4814

    2022

  30. [30]

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Frozen clip models are efficient video learners. In European Conference on Computer Vision . Springer, 388–404

    2022

  31. [31]

    In Proceedings of the IEEE/CVF International Conference on Com- puter Vision

    Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision. 13536–13545

  32. [32]

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860 (2021)

    Pith/arXiv arXiv 2021

  33. [33]

    Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. 2020. Something-else: Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 1049–1059

    2020

  34. [34]

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 3202–3211

    2022

  35. [35]

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jian- long Fu, Shiming Xiang, and Haibin Ling. 2022. Expanding language-image pretrained models for general video recognition. In European conference on com- puter vision. Springer, 1–18

    2022

  36. [36]

    Yangjun Ou, Li Mi, and Zhenzhong Chen. 2022. Object-relation reasoning graph for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 20133–20142

    2022

  37. [37]

    Yujie Mo, Yajie Lei, Jialie Shen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. 2023. Disentangled multiplex graph representation learning. InInternational conference on machine learning . PMLR, 24983–25005

    2023

  38. [38]

    Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui

  39. [39]

    Zefeng Qian, Xincheng Yao, Yifei Huang, Chongyang Zhang, Jiangyong Ying, and Hong Sun. 2025. Beyond label semantics: Language-guided action anatomy for few-shot action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 10421–10431

    2025

  40. [40]

    Junting Pan, Siyu Chen, Mike Zheng Shou, et al . 2021. Actor-context-actor relation network for spatio-temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 464–474

    2021

  41. [41]

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Fine-tuned clip models are efficient video learn- ers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6545–6554

    2023

  42. [42]

    Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14 . Springer, 510–526

    2016

  43. [43]

    Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional net- works for action recognition in videos. Advances in neural information processing systems 27 (2014)

    2014

  44. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PmLR, 8748–8763

    2021

  45. [45]

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri

  46. [46]

    Heng Wang and Cordelia Schmid. 2013. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision . 3551–3558

    2013

  47. [47]

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. 2024. Omnivid: A generative framework for universal video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18209–18220

    2024

  48. [48]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35 (2022), 10078–10093

    2022

  49. [49]

    Ning Wang, Guangming Zhu, HS Li, Liang Zhang, Syed Afaq Ali Shah, and Mohammed Bennamoun. 2024. Language Model Guided Interpretable Video Action Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18878–18887

    2024

  50. [50]

    Xiaolong Wang and Abhinav Gupta. 2018. Videos as space-time region graphs. In Proceedings of the European conference on computer vision (ECCV) . 399–417

    2018

  51. [51]

    Yunbo Wang, Mingsheng Long, Jianmin Wang, and Philip S Yu. 2017. Spatiotem- poral pyramid network for video action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition . 1529–1538

    2017

  52. [52]

    T. Wu, R. He, G. Wu, et al. 2024. Sportshhi: A dataset for human-human interaction detection in sports videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18537–18546

    2024

  53. [53]

    M Wang, J Xing, and Y Liu. [n. d.]. Actionclip: A new paradigm for video action recognition. arXiv 2021. arXiv preprint arXiv:2109.08472 ([n. d.])

    Pith/arXiv arXiv 2021

  54. [54]

    Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV). 305–321

    2018

  55. [55]

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. 2022. Clip-vip: Adapting pre-trained image-text model to video- language representation alignment. arXiv preprint arXiv:2209.06430 (2022)

    Pith/arXiv arXiv 2022

  56. [56]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)

    Pith/arXiv arXiv 2022

  57. [57]

    Boqiang Zhang, Hongtao Xie, Zuan Gao, and Yuxin Wang. 2024. Choose what you need: Disentangled representation learning for scene text recognition removal and editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 28358–28368

    2024

  58. [58]

    Tianci Wu, Guangming Zhu, Jiang Lu, Siyuan Wang, Ning Wang, Nuoye Xiong, and Zhang Liang. 2025. Prompt-guided Disentangled Representation for Action Recognition. arXiv preprint arXiv:2509.21783 (2025)

    arXiv 2025

  59. [59]

    Jingran Zhang, Fumin Shen, Xing Xu, and Heng Tao Shen. 2020. Temporal reasoning graph for activity recognition. IEEE Transactions on Image Processing 29 (2020), 5491–5506

    2020

  60. [60]

    Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees GM Snoek. 2022. Audio- adaptive activity recognition across video domains. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 13791–13800

    2022

  61. [61]

    Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. 2019. A structured model for action detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9975–9984

    2019

  62. [62]

    Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid. 2023. How can objects help action recognition?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2353–2362

    2023

  63. [63]

    Ce Zhang, Changcheng Fu, Shijie Wang, Nakul Agarwal, Kwonjoon Lee, Chiho Choi, and Chen Sun. 2024. Object-centric video representation for long-term ac- tion anticipation. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 6751–6761

    2024

  64. [64]

    Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan Kankan- halli. 2019. Explainable video action reasoning via prior knowledge and state transitions. In Proceedings of the 27th acm international conference on multimedia . 521–529. Knowledge-guided Disentanglement with Atomic Actions for Action Recognition MM ’26, November 10–14, 2026, Rio de Ja...

    2019

  65. [68]

    Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. 2020. S3vae: Self-supervised sequential vae for representation disentanglement and data gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6538–6547

    2020

  66. [70]

    For example, a single head dominates the Top 10% activations (62.49%), while most other heads are rarely se- lected

    Under-capacity (𝑀≤ 8): When𝑀 = 8, the routing distribu- tion is highly imbalanced. For example, a single head dominates the Top 10% activations (62.49%), while most other heads are rarely se- lected. This indicates a capacity bottleneck, where multiple seman- tic factors are forced to share limited representational components, leading to strong competitio...

  67. [71]

    Under a strict threshold (Top 10%), only a subset of heads is frequently activated, Table 8: Ablation study of different numbers of disentanglement heads in KDM

    Balanced capacity ( 𝑀 ∈ [10, 12]): When 𝑀 = 10 or 12, the routing distribution becomes more structured. Under a strict threshold (Top 10%), only a subset of heads is frequently activated, Table 8: Ablation study of different numbers of disentanglement heads in KDM. We accordingly adjust the number of learnable se- mantic anchors to match the number of hea...

  68. [72]

    This may be attributed to over-parameterization, where the routing distribution becomes more diffuse and some heads are under-utilized

    Over-capacity (𝑀 > 12): When further increasing𝑀 (e.g., 𝑀 = 14), the performance slightly degrades. This may be attributed to over-parameterization, where the routing distribution becomes more diffuse and some heads are under-utilized. Such redundancy can weaken the discriminative power of individual components and reduce overall efficiency. Overall, thes...

    2026

  69. [2015]

    Learning spatiotemporal features with 3d convolutional networks. In MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Tianci wu, Siqi Cao, Guangming Zhu, Jiang Lu, Siyuan Wang, Longfei Zhang, Jincai Huang, Jun Sheng, and Liang Zhang Proceedings of the IEEE international conference on computer vision . 4489–4497

    2026

  70. [2018]

    Advances in neural information processing systems 31 (2018)

    Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems 31 (2018)

    2018

  71. [2021]

    Advances in Neural Informa- tion Processing Systems 34 (2021), 21872–21884

    Disentangled contrastive learning on graphs. Advances in Neural Informa- tion Processing Systems 34 (2021), 21872–21884

    2021

  72. [2022]

    arXiv preprint arXiv:2207.07646 (2022)

    Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint arXiv:2207.07646 (2022)

    Pith/arXiv arXiv 2022