REVIEW 4 major objections 6 minor
Knowledge-guided Disentanglement with Atomic Actions for Action Recognition
T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read The paper claims that LLM-generated atomic actions — fine-grained spatial-temporal descriptions of each action label — injected into scene-graph video features improve multi-label action recognition beyond prompt-only disentanglement.
desk verdict The oracle numbers rest on an undefined test-time Action Specification, and the standard Charades row trails ProDA; the idea is worth engaging, but the SOTA claim doesn't hold as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Atomic Action Memory Bank: each action label is decomposed by an LLM into temporally ordered spatial-temporal descriptions, encoded by a text encoder, and stored per-step in memory banks. A scoring network with a straight-through estimator thresholds a variable number of atomic entries per scene-graph node, which is how overlapping actions share nodes without node-level supervision. In KDM the same selection is masked by the Action Specification (the multi-hot SAP/UAP pair), so only knowledge relevant to specified or residual actions is retrieved; learnable embeddings fused alongside buffer the semantics so they do not dominate. The Knowledge Disentanglement Lo
What would settle it
A controlled run of KDA on Charades using oracle scene graphs but with SAP/UAP prompts built from the model's own predictions (or from a single fixed full-label prompt) rather than ground-truth labels: if mAP falls from 73.2 toward the standard-setting level around 50, the gains attributed to atomic-action knowledge are driven by label identity leaking through the prompts rather than by the semantic descriptions themselves.
Extended reading notes
Core claim
On its own terms, the paper finds that injecting fine-grained semantic knowledge into scene graphs, then re-disentangling it under action-specification prompts, yields more discriminative features for multi-label action recognition. The central discovery is a balance: atomic actions alone inject overly strong semantics that overwhelm visual features (69.35 mAP), label-level knowledge is too coarse (70.96), and only atomic actions combined with learnable embeddings reach the best result (73.2 mAP under oracle scene graphs vs 71.1 for the prior prompt-only method). A margin-bounded knowledge-disentanglement loss over parallel heads keeps components mutually independent yet aligned to semantic
Load-bearing premise
The load-bearing premise is that the Action Specification prompts steering disentanglement can be built at test time without ground-truth labels: the supplementary material constructs SAP/UAP prompts from ground-truth label sets, and in the standard protocol, where scene graphs are predicted, KDA's 49.9 mAP trails the prompt-only baseline's 50.2 — so the reported headline gains rest on this unstated prompt-construction assumption.
Editorial extensions
If this is right
- Adding KIM and KDM to existing action-recognition models improves mAP and recall (the paper reports gains on SlowFast and the SportsHHI baseline), so fine-grained knowledge injection transfers across backbones.
- Atomic action guidance helps most for coverage-sensitive metrics like R@20/R@50 and for classes absent from training, so the benefit should grow when the label space is broad or open.
- The gains do not depend on a specific LLM: comparable results across three LLMs and structured prompt templates show the knowledge source is interchangeable, while an unstructured template is consistently worse.
- Even when scene-graph relationship annotations are removed, atomic actions supply enough relational information to stay near the full-graph performance (70.9 vs 73.2 mAP), suggesting the semantic descriptions partially replace structured annotations.
Reading between the lines
- Editorial inference: the headline comparison is an oracle setting — both scene graphs and the Action Specification prompts are built from ground-truth labels; under the standard protocol KDA's 49.9 mAP sits below the prior prompt-only method's 50.2. A direct test is to build test-time SAP/UAP prompts from the model's own predictions or a fixed prompt and measure how much of the oracle gain survive
- Editorial inference: the retrieve–inject–disentangle pattern could generalize to other tasks with text-decomposable factors (group activities, human-object interactions, assembly or surgical steps), provided the knowledge strength is calibrated so the semantics do not overwhelm the visual encoder.
- Editorial inference: the reported head-routing statistics point to capacity allocation as the performance ceiling; a dynamic head-count or per-node head selection mechanism could avoid the under-capacity (8 heads) and over-capacity (14 heads) regimes the paper observes.
- Editorial inference: the ablations show more semantic detail is not uniformly better — atomic actions alone hurt — so an explicit search over knowledge granularity (number of atomic steps, sentence vs. phrase level) would map the sweet spot the paper identifies only qualitatively.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KDA, a framework for multi-label action recognition that uses LLM-generated 'atomic actions' to inject fine-grained semantic knowledge into spatial-temporal scene graph features (KIM) and to condition a prompt-based disentanglement module (KDM) via a Specified/Unspecified Action Prompt pair. A novel Knowledge Disentanglement Loss regularizes the disentanglement heads. Experiments on Charades and SportsHHI report gains over ProDA in the oracle Charades setting (73.2 vs 71.1 mAP) and on SportsHHI (11.70 vs 10.69 mAP), but the standard Charades result is 49.9 mAP versus ProDA's 50.2.
Significance. The core idea of using LLM-generated atomic actions to provide explicit semantic guidance for action disentanglement is timely and potentially useful; the hierarchical feature/structure/semantic design is a natural extension of ProDA. The paper includes several ablations, an efficiency analysis, and a parameter-matched comparison on SportsHHI, which are helpful. However, the paper's central claim of state-of-the-art performance is currently not supported by the non-oracle experiments, and a load-bearing protocol detail—how Action Specifications are constructed at test time—is left unspecified. If the authors can clarify and fix that protocol, the contribution may be valuable; in its present form the empirical evidence is inconclusive.
major comments (4)
- [§4.2, §A.1–A.2, Eq. (6), Eq. (10)] The test-time construction of the Action Specification is never defined. The supplementary builds SAPs from ground-truth label sets (Eqs. 16–25) and UAP as their complement, and KDM consumes this AS at inference through Eq. (6) and Eq. (10). If test-time AS is constructed the same way, the oracle Charades gain (73.2 vs ProDA 71.1) is confounded by label leakage: the model is handed the set of present actions before classification. The only non-leak comparison is the standard Charades row in Table 1, where KDA (49.9) trails ProDA (50.2). Please specify how p_sap and p_uap are obtained at test time without ground-truth labels, and report results under that protocol.
- [Abstract, §1, Table 1] The abstract and contributions claim state-of-the-art performance on multi-label action recognition benchmarks. Table 1 shows KDA under the standard protocol at 49.9 mAP, below ProDA at 50.2, and the text itself describes this as 'comparable.' The SOTA claim is therefore supported only by the oracle row, which uses ground-truth scene graphs and, per the supplementary, ground-truth labels in the AS. This overstates the evidence. Revise the claim or provide a standard-protocol result with a positive margin; additionally, note that the oracle and standard rows use different visual features (CLIP vs LaIAR), further complicating cross-setting inference.
- [§3.2–§3.4, Tables 1–2] Reproducibility and uncertainty: the main results are single runs without error bars, and the key differences are small relative to typical run-to-run variance (49.9 vs 50.2 on Charades standard; +1.01 mAP on SportsHHI). Hyperparameters introduced in Section 3—τ, M, L, t, γ, λ1, λ2, m1, m2—are not reported; only M and L are ablated in the supplementary. The paper states 'The code is available here' but no URL is given. These omissions prevent independent verification of the central empirical claim.
- [§2.2, §3.1, Table 4] The atomic actions are generated from the supervised action labels via an LLM, so the injected 'knowledge' is semantically derived from the prediction target itself. This does not invalidate the approach, but the paper should frame the contribution as a richer conditioning signal rather than independent external knowledge. Table 4 compares label versus atomic-action knowledge in the oracle setting; a standard-setting version of this comparison and a parameter-matched label-only baseline would strengthen the claim that gains come from atomic-action granularity rather than model capacity.
minor comments (6)
- [Figure 1] The caption lists parts (a) and (b), but the text in §1 and §3.1 refers to 'Fig. 1(c)' twice. Either add panel (c) to the figure/caption or fix the references.
- [Eq. (14) vs Supplementary Eq. (29)] The KD Loss in the supplementary (Eq. 29) writes h_m^* in the denominator while the main text (Eq. 14) uses h_m. Clarify whether these are the same quantity or whether normalization is intended; as written, the two equations are inconsistent.
- [Table 1] The caption defines 'SSG: ground-truth SSG' which is tautological; presumably SSG stands for 'scene graph.' Also define 'Bbox' and clarify which rows use predicted versus ground-truth scene graphs.
- [Table 2] Rows labeled 'SportsHHI (V)[53]' and 'SportsHHI (V)[52]' are confusing: [53] is the ProDA reference, not a SportsHHI baseline variant. Clearly identify the backbone and source for each row.
- [Author affiliations and code link] Typos: 'Human Institute of Advanced Technologyy' and lowercase 'Tianci wu' should be fixed. The phrase 'The code is available here' should include a working URL.
- [Table 5] The text claims that the method 'attains the highest minimum accuracy,' but the table reports only mean accuracy and standard deviation. Add a minimum-accuracy column or remove the claim.
Circularity Check
SOTA claim rests on unstated test-time Action Specification: §A builds SAP/UAP from ground-truth labels, and standard Charades (49.9) trails ProDA (50.2).
-
other
[Supplementary §A.1–A.2; §3.3 Eq. (6), Eq. (10); Table 1]
"Given a video with ground-truth label set{0, 1}, its multi-hot representation is [1, 1, 0, 0, 0]. ... Based on the ground-truth labels, we first enumerate all non-empty subsets of the label set. ... Given a video with ground-truth label set {77, 79, 80, 76, 75}, present labels are sampled from this set, while absent labels are sampled from the remaining label space... To prevent information leakage, the corresponding UAP is defined as the complement of each SAP in the full label space."
The only Action Specification protocol given in the paper is built from ground-truth labels: SAP is a multi-hot vector of sampled true labels and UAP is its complement. KDM then conditions everything on this AS via Eq. 6 (˜δ = δ ⊙ p_as) and Eq. 10 (ˆf_j_{i,as} = MLP([f_j_i; g_as])). If this same construction is used at oracle evaluation — and the paper never specifies any other test-time AS protocol — the model is handed the true label set as an input before classifying those labels. The oracle Charades gain over ProDA (73.2 vs 71.1) is therefore at least partly attributable to label leakage rather than to atomic-action disentanglement. The non-oracle Charades row (49.9) is below ProDA (50.2), so the abstract's 'state-of-the-art' claim rests on this unspecified, leak-prone setting.
full rationale
The paper's derivation chain does not contain a fitted parameter that is renamed as a prediction, and there is no imported uniqueness theorem from the authors' prior work. The atomic actions are generated from action labels, but using label semantics as text input is a standard class-name-conditioning strategy and is not itself circular. The self-citations to ProDA [53] are architectural (VGPNN, AD Loss, random-sampling AS) rather than load-bearing external evidence; the central comparison is on held-out benchmark splits. The main circularity-adjacent issue is the Action Specification protocol: the supplement builds SAP/UAP exclusively from ground-truth label sets, while the main text never states how AS is produced at test time. Since Eq. 6 and Eq. 10 feed AS directly into the disentanglement and guided feature computation, an oracle evaluation that reuses the §A construction would put the prediction target into the input. This is a missing protocol step with direct potential for label leakage, not a fully demonstrated reduction by construction; the standard Charades result also trails ProDA, so the SOTA claim is fragile. Weighing these, the paper is only partially circular: score 3.
Assumptions & free parameters
free parameters (7)
- threshold tau =
not reported
- AD margins m1, m2 =
not reported
- loss weights lambda1, lambda2 =
not reported
- KD temperature t =
not reported
- KD margin gamma =
not reported
- disentanglement heads M / semantic anchors L =
M=10, L=64
- LLM and prompt template =
Qwen3-plus + ST prompt (main paper)
assumptions (6)
- domain assumption LLM-generated atomic action descriptions from labels are semantically faithful and useful for visual action recognition.
- domain assumption CLIP's text embedding space aligns atomic-action text with SSG node visual features.
- standard math Straight-through estimator gradient surrogate for discrete selection is adequate.
- domain assumption SAP/UAP decomposition with distractor labels prevents information leakage and supports multi-label disentanglement at test time.
- domain assumption ProDA's VGPNN and AD loss (Pearson correlation + reconstruction) provide a valid disentanglement backbone.
- domain assumption SSG construction (predicted or oracle) provides sufficient relational signal for atomic-knowledge injection.
invented entities (2)
-
Atomic Action Memory Banks (AMBs)
-
Learnable semantic anchors U
Cite this review
Pith. "Pith review of Knowledge-guided Disentanglement with Atomic Actions for Action Recognition." pith.science (2026). https://pith.science/paper/QSF32PLD
@misc{pith2026260726097,
author = {Pith},
title = {Pith review of: Knowledge-guided Disentanglement with Atomic Actions for Action Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/QSF32PLD}},
note = {Machine review of arXiv:2607.26097}
}
read the original abstract
Action recognition in complex scenes often involves multiple concurrent fine-grained actions, making it challenging to model internal action structures. Most existing methods rely on holistic representations, which are insufficient for capturing subtle interactions and fine-grained semantics. While recent prompt-based approaches introduce disentanglement, they lack explicit semantic guidance, and methods based solely on visual or structured cues remain coarse-grained. In this paper, we propose Knowledge-guided Disentanglement with Atomic Actions (KDA), which leverages fine-grained semantic knowledge to enhance action representations and enable more precise disentanglement. Specifically, we use Large Language Models (LLMs) to decompose action labels into atomic actions, providing explicit spatial-temporal semantics. A Knowledge Injection Module (KIM) first integrates atomic action knowledge into video features. Based on this enhanced representation, a Knowledge Disentanglement Module (KDM) further disentangles atomic action knowledge to produce more precise semantic guidance for action disentanglement. A Knowledge Disentanglement Loss (KD Loss) is introduced to encourage clearer disentanglement of knowledge components within KDM. Extensive experiments demonstrate that KDA improves feature discriminability and achieves state-of-the-art performance on multi-label action recognition benchmarks. Moreover, KIM and KDM can be readily integrated into other methods, demonstrating strong generality.
Figures
Figures from the paper (2 more)
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.