{"id":"37c44ea5-3a77-49c4-bba3-b2a932fe99b8","arxiv_id":"2501.05066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"By adding detected object nodes to skeleton graphs and training with a variable-graph network plus a random node attack regularizer, the authors report large accuracy gains over skeleton-only baselines on action recognition benchmarks.","lead":"This paper adds information about the objects a person uses, such as books and tools, into skeleton-based action recognition networks, and reports accuracy gains on standard benchmarks. A generalist should care because object interactions often separate actions that look nearly identical in skeleton data, and the approach targets privacy-friendly industrial monitoring.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The self-training split for object-node annotation is unspecified, so the reported NTU60 gains could be inflated if test frames entered the detector's pseudo-label pool; this needs a stated split or rerun before the SOTA claim is accepted.","rationale":"The central claim requires that the object-node annotations are both informative and generated without using the action-recognition test videos. The reader identified this as the weakest assumption, and I agree: Algorithm 1 leaves the unlabeled pool Dp unspecified, so there is no way to tell whether the object detector saw test frames during self-training. That is a real correctness risk because the detector's outputs are inputs to the action model at test time. The paper's architecture and ablations are plausible, and the framework has internal consistency, but the headline NTU60 numbers cannot be independently verified from the preprint alone: code, data, and detector weights are not provided. The JXGC 24 sample-count inconsistency reinforces the need for release of exact dataset metadata. I am not moving the verdict because the concern is conditional rather than a demonstrated error: if the authors disclose that Dp is train-only, or if a clean rerun reproduces the numbers, the claim stands. The NTU120 results are not state-of-the-art, but the paper's strongest claim is about NTU60 and JXGC, so that is not the primary blocker. Thus the existing CONDITIONAL verdict is appropriate, and no adjustment is needed.","tokens_in":14199,"tokens_out":4994,"duration_ms":54370,"concrete_test":"Rerun the NTU60 experiments with the object detector trained only on the action-recognition training split: explicitly specify Dp by subject/camera IDs, or set Dp to be disjoint from the X-Sub and X-View test videos, then regenerate all test object nodes and re-evaluate Table 2. If X-Sub/X-View accuracy drops materially (e.g., more than 0.5%) from the reported 96.7/99.2, the reported object-node gains are inflated by transductive self-training; if accuracy is unchanged, the concern is resolved. Releasing the object-node annotations for both splits would allow an independent check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.2 and Algorithm 1 describe a self-training object detector that produces the roughly 2 million object nodes, but the composition of Dl and Dp is never stated. The central NTU60 numbers (96.7/99.2) therefore rest on annotations whose generation process could include the test frames: if Dp contains test-set videos, the detector is trained or pseudo-labeled on those frames, and the object nodes fed to ST-VGCN at test time are not produced by a detector that has only seen the training distribution. This would inflate the measured contribution of object nodes over skeleton-only baselines. The paper does not release the object annotations, detector weights, or enough code to verify the split, and Table 1's JXGC sample count (3,174) conflicts with Sec. 4.1.3 (1,770), further weakening independent audit. The concern is not an allegation of misconduct, but the presented evidence cannot rule out a test-influenced annotation pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ST-VGCN, a spatial-temporal variable graph convolutional network for skeleton-based action recognition that adds object nodes encoding position, detection probability, and CLIP text-based class attributes to the skeleton graph. It introduces variable graph construction to handle sequences with variable node counts, and four components: Class Attribute Fusion (CAF), Weighted Node Pooling (WNPool), Node Balance Loss, and Random Node Attack. To obtain object annotations, the authors employ a self-training YOLOv5l detector, producing roughly 2 million object nodes and creating two datasets: NTU RGB+D+Object 60 and JXGC 24. The paper reports state-of-the-art accuracy on NTU RGB+D 60 (96.7% cross-subject, 99.2% cross-view) and large gains on JXGC 24.","tokens_in":14407,"tokens_out":7788,"duration_ms":69004,"significance":"If the object annotations are generated without test-set leakage, the paper would make a substantive contribution: it shows that object category and position information can substantially improve skeleton-based action recognition, it introduces a new annotated dataset and an object-augmented version of NTU60, and it addresses the resulting object-category shortcut with a dedicated augmentation. The ablations are systematic, and the module-by-module analysis (Tables 4 and 6) is useful. However, the validity of the headline numbers depends entirely on the self-training annotation pipeline, whose train/test separation is not described; the reported numbers should not be used to compare against prior work until this is resolved. Reproducibility is also weakened by the dataset-count inconsistency and the lack of object-detection accuracy reporting on the pseudo-labels.","major_comments":[{"comment":"The composition of the labeled set D_l and the pseudo-labeled set D_p in the self-training object detector is never stated. In particular, the paper does not say whether D_p includes any videos from the NTU RGB+D 60 test splits that are later used to evaluate ST-VGCN. If test frames are included in D_p, the object detector would have been trained or pseudo-labeled on those frames, so the object nodes fed to the action recognizer at test time are not produced by a model blind to the evaluation set. This would directly inflate the reported gains (3.3% on X-Sub, 1.2% on X-View) and would make the comparison against prior skeleton-only methods unfair. The authors must state the exact split used for D_l and D_p, and ideally verify that all test-set videos are excluded from D_p; alternatively, they should rerun the experiments with a detector trained only on training-set frames.","section":"Section 3.1.2, Algorithm 1"},{"comment":"The JXGC 24 dataset is described as containing 1,770 video samples in Section 4.1.3, but Table 1 lists 3,174 samples. This is a direct factual contradiction in the dataset description. Please correct the number, state which value is actually used in the experiments of Tables 2 and 3, and check whether the dataset composition described elsewhere (8 object types, 3 actions, 24 combined classes) is consistent with the reported count.","section":"Section 4.1.3 and Table 1"},{"comment":"No evaluation of the object detector used for pseudo-labeling is reported. The paper gives no mAP or accuracy for the YOLOv5l self-training process, nor any measure of noise in the approximately 2 million auto-generated object nodes. Since the central improvement is attributed to these object nodes, the reader cannot judge whether the reported gains are robust to detector errors. Please report detection performance on a separated validation set and, if possible, analyze the sensitivity of the action-recognition results to pseudo-label noise.","section":"Section 3.1.2 and Experiments"}],"minor_comments":[{"comment":"The text says combining position and class attributes yields a 3.5% improvement, but Table 4 shows 3.9% (96.1 vs. 92.2). Please correct the inconsistency.","section":"Section 4.3.1"},{"comment":"The statement that \"the network achieves optimal classification performance 93.0% when the parameter λ is set to 0.1\" is ambiguous; it should state the split (presumably JXGC 24 \"Both\") and be reconciled with the 96.1% final accuracy reported in Table 6.","section":"Section 4.3.5"},{"comment":"The loop condition \"losscount − losscount−1 > c\" is undefined at the first iteration because losscount−1 is not initialized. Please clarify the convergence criterion.","section":"Algorithm 1"},{"comment":"There are typos (\"designe\", \"Aandom\"), and the phrase \"directional interference\" should be defined or replaced with a concrete description of the attack.","section":"Abstract and Section 4.3.6"},{"comment":"The formatting of the last rows is garbled (e.g., \"MOurs\" and \"T able\") and several entries are concatenated (e.g., \"91.91.8 -\"); please ensure the table is readable and that all numbers are correctly aligned.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concerns are valid: the self-training split is unspecified and the JXGC sample count is inconsistent. In my reading, the Random Node Attack experiment, while showing sensitivity to object category, actually supports the paper's own claim that object category is a shortcut; the proposed mitigation is reasonable, but it does not address the leakage question. I would recommend requiring the split information or a rerun before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-written paper with a plausible core idea — add object nodes to skeleton graphs and model the variable graph — and the ablations are internally consistent. But the headline NTU60 numbers rest on a self-training pipeline whose train/test separation is never described, and there's a concrete sample-count inconsistency that makes me wary. I'd send it to review, but I'd insist on a stated split and a rerun or release before believing the 96.7/99.2.\n\nWhat's actually new: the variable graph construction (handles a changing number of object nodes per frame) and Random Node Attack, a data augmentation that injects random object nodes to prevent the model from latching onto category alone. That augmentation is a good idea, and Table 7 shows it works: without it, inserting a 'Book' node nukes accuracy to 58.7; with it, 95.8. That's compelling evidence of the failure mode and the fix. The CLIP-based class attributes and Node Balance Loss are minor but sensible. JXGC 24, though small, is a useful industrial benchmark with a clean three-way split (action, workpiece, both).\n\nThe soft spots are real. The big one is Section 3.1.2 and Algorithm 1: the self-training object detector is described, but Dl and Dp are never defined. If Dp includes test frames, the object nodes at test time come from a detector that has seen the test videos, which would inflate the reported gains. I'm not alleging misconduct; I'm saying the paper as written doesn't rule it out. That's load-bearing because the entire contribution is the object-node signal. Also, Table 1 says JXGC 24 has 3,174 samples while Section 4.1.3 says 1,770; those can't both be right. Minor, but it undermines confidence in the numbers. On NTU120, the method doesn't beat prior SOTA, so the abstract's 'surpasses the previous state-of-the-art on multiple benchmarks' is oversold — it's SOTA on two NTU60 splits only. A GitHub link appears, but no statement that code or data are released, so independent verification is currently impossible.\n\nWho is this for? People working on skeleton-based action recognition, especially human-object interaction. The idea is worth engaging with, and Random Node Attack is a practical trick worth stealing. But I wouldn't take the NTU60 numbers at face value until the self-training split is stated and the data inconsistency fixed.\n\nRecommendation: send to peer review with a major-revision request: state the split, correct the sample count, release the annotations or at least the detection pipeline, and temper the SOTA claim.","headline":"Plausible idea with a real SOTA claim on NTU60, but the unspecified self-training split and an internal data inconsistency keep me from trusting the numbers as-is.","tokens_in":14951,"tokens_out":3405,"would_cite":false,"duration_ms":29557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Detected interactive objects inserted into the skeleton graph push NTU RGB+D 60 accuracy to 96.7% cross-subject and 99.2% cross-view, surpassing prior state of the art.","keywords":["graph convolutional networks","skeleton-based action recognition","human-object interaction","variable graph","object detection pseudo-labels","data augmentation","NTU RGB+D","JXGC 24"],"falsifier":"Train the same ST-VGCN with object nodes generated by a detector trained only on the official training split of NTU RGB+D 60, with all validation and test videos excluded from the unlabeled pool, and rerun the X-Sub and X-View benchmarks; if the reported 96.7% and 99.2% cannot be reproduced without test-frame leakage, the claimed gains are not from object information alone. A simpler check is to count how often the detected object category on test frames matches the ground-truth action category for actions like reading, and whether removing those object categories from the annotation changes accuracy.","tokens_in":14000,"feed_emoji":"🦴","tokens_out":7747,"duration_ms":67655,"temperature":0.7,"pith_summary":"This paper argues that the reason skeleton-only action recognition confuses actions such as reading and writing is that the skeleton graph carries no information about the objects a person touches. To fix this, it adds detected interactive objects as extra graph nodes, each carrying position, detection confidence, and a text-derived class embedding, and builds a Spatial Temporal Variable Graph Convolutional Network (ST-VGCN) that can process a variable number of nodes per frame. The authors construct two object-enriched datasets (NTU RGB+D+Object 60 and JXGC 24) with more than two million object-node annotations and report state-of-the-art accuracy, 96.7% on NTU RGB+D 60 cross-subject and 99.2% on cross-view. A sympathetic reader would care because the gains come from a cheap modality, object detection, applied to a privacy-friendly skeleton representation, and because the variable-graph machinery removes a fixed-topology constraint of prior GCNs.","feed_headline":"Object nodes push skeleton action accuracy to 96.7% and 99.2%","feed_subtitle":"Adding object positions, categories, and a variable-node graph beats prior skeleton-only models on both NTU RGB+D 60 splits.","key_machinery":"The load-bearing object is the Variable Graph (VG), built under the rule 'one frame, one graph': every frame contributes all detected skeletons and all detected objects as a single directed node set, so the graph size varies across frames and videos. A second mechanism is the object node itself, whose three concatenated parts, position, detector probability, and a CLIP text-encoded class attribute, carry the interactive information into the network. ST-VGCN then handles the variable size with node padding and keeps object information from dominating through Class Attribute Fusion, Weighted Node Pooling, Node Balance Loss, and Random Node Attack augmentation. The unidirectional object-to-skeleton edge is what prevents padded empty nodes from turning into noise during graph convolution.","core_discovery":"On its own terms, the paper's discovery is that interactive objects are not auxiliary context but a missing part of the action's state, and that injecting them into the skeleton graph as additional nodes is enough to surpass prior skeleton-based state of the art. The object node feature is the concatenation of object position, detector probability, and a CLIP-encoded class attribute; each frame becomes a directed Variable Graph with all skeleton joints of all people plus all object nodes, with skeleton-skeleton edges in both directions and object-to-skeleton edges in one direction. ST-VGCN is an ST-GCN-style network augmented with node padding for variable graph sizes, a Class Attribute Fusion module, weighted node pooling, a Node Balance Loss that keeps skeleton and object channels in proportion, and a Random Node Attack augmentation that inserts fake object nodes during training to prevent the classifier from latching onto object identity. With these pieces, the method reports 96.7% (X-Sub) and 99.2% (X-View) on NTU RGB+D 60, beating the strongest cited prior result by 2.6 and 1.4 percentage points, and 91.4% on the new JXGC 24 'Both' split, 8.1 points above its own no-object baseline.","pith_inferences":["Beyond the paper, if the pseudo-label pipeline is sound, the same object-node recipe could be applied to other skeleton benchmarks without manual annotation, since the object detector only needs RGB frames and the method already works with 2D pose.","Beyond the paper, the variable-graph construction is not limited to objects; the paper's own conclusion suggests extending it to human-background and object-object relations, which would let one GCN ingest heterogeneous scene information under the same padding scheme.","Beyond the paper, a natural test is whether Random Node Attack can serve as a general regularizer for any multimodal graph where one modality's identity features are much stronger than another's, not just object-versus-skeleton imbalance.","Beyond the paper, because the NTU RGB+D 120 evaluation reuses object annotations from the first 60 classes, one would expect the advantage to shrink on the 60 new classes; the paper's reported 120-level gains of 1.5 to 1.7 points are an upper bound for generalization to unseen objects."],"forward_implications":["Actions that look nearly identical in joint coordinates, such as reading versus writing, become separable once a book or pen node is present.","The same fixed skeleton backbones can be upgraded by appending object nodes, at small cost: parameter count rises only from 2.71M to 2.72M and inference time from 6.43 ms to 6.73 ms in the reported setting.","Recognition in industrial settings can be organized around two independent dimensions, what action and which workpiece, and object nodes improve both, with the largest jump on workpiece classification (99.8% vs. 93.0% for the strongest prior baseline in the 'Workpiece' split).","Random Node Attack makes the model robust to spurious objects: without it, adding a book node to an unrelated video collapses accuracy to 58.7%; with it, accuracy remains at 95.8%."],"supporting_citations":[{"why":"ST-GCN is the backbone architecture that ST-VGCN minimally modifies, so the object-node gains are measured against this baseline.","marker":"[5]"},{"why":"The toolkit that supplied HRNet-estimated 2D skeletons and the comparative implementations used for the baseline results.","marker":"[6]"},{"why":"Prior object-node approach using contour keypoints and category index, which the paper contrasts with its position-plus-class-attribute object nodes.","marker":"[15]"},{"why":"Earlier method that detects object areas by subtracting human areas, providing the existing object-information baseline the paper extends.","marker":"[17]"},{"why":"CLIP text encoder is used to pre-embed textual descriptions of object categories into class attribute vectors.","marker":"[19]"},{"why":"NTU RGB+D is the base dataset that the paper extends with roughly two million object nodes to create NTU RGB+D+Object 60.","marker":"[20]"},{"why":"The pseudo-label self-training approach that generates the large object-node annotation set with limited manual labeling.","marker":"[34]"},{"why":"HRNet pose estimator that produces the 2D skeleton coordinates used across all datasets and baselines.","marker":"[43]"}],"fun_headline_variants":["Object nodes push action recognition past prior skeleton SOTA","Interactive object info boosts skeleton action recognition","New framework uses objects to hit 99.2% on action recognition","ST-VGCN: combining skeleton and objects for record accuracy","Object-aware graph network surpasses skeleton-only benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the roughly two million object-node annotations produced by the self-trained detector are accurate and that the detector never saw test-set frames while being trained or pseudo-labelled; the paper does not specify which videos formed the unlabeled data set, so this contamination can neither be confirmed nor ruled out.","fun_headline_variants_meta":{"raw":{"variants":["Object nodes push action recognition past prior skeleton SOTA","Interactive object info boosts skeleton action recognition","New framework uses objects to hit 99.2% on action recognition","ST-VGCN: combining skeleton and objects for record accuracy","Object-aware graph network surpasses skeleton-only benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1542,"prompt_tokens":1103,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":719,"tokens_out":439,"duration_ms":4839,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:19:28.729503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same ST-VGCN with object nodes generated by a detector trained only on the official training split of NTU RGB+D 60, with all validation and test videos excluded from the unlabeled pool, and rerun the X-Sub and X-View benchmarks; if the reported 96.7% and 99.2% cannot be reproduced without test-frame leakage, the claimed gains are not from object information alone. A simpler check is to count how often the detected object category on test frames matches the ground-truth action category for actions like reading, and whether removing those object categories from the annotation changes accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The toolkit that supplied HRNet-estimated 2D skeletons and the comparative implementations used for the baseline results."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"Prior object-node approach using contour keypoints and category index, which the paper contrasts with its position-plus-class-attribute object nodes."},{"cited_title":"In: Meila, M., Zhang, T","cited_arxiv_id":null,"evidence_quote":"CLIP text encoder is used to pre-embed textual descriptions of object categories into class attribute vectors."},{"cited_title":"In: Workshop on Challenges in Representation Learning, ICML, vol","cited_arxiv_id":null,"evidence_quote":"The pseudo-label self-training approach that generates the large object-node annotation set with limited manual labeling."},{"cited_title":"HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation","cited_arxiv_id":"1908.10357","evidence_quote":"HRNet pose estimator that produces the 2D skeleton coordinates used across all datasets and baselines."}],"review_version":1}