{"id":"5d710717-20e4-4cc6-aea1-c1e522c04670","arxiv_id":"2607.17342","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Skeleton-only interaction recognition that aligns skeleton and video features during training outperforms prior state-of-the-art on four benchmarks.","lead":"STAR is a method for recognizing human-human and human-robot interactions from skeleton data alone, while using RGB video hints only during training. It reports accuracy gains over prior methods on four interaction recognition benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed visual-alignment benefit is small and statistically unsecured: the skeleton encoder alone (91.50) already beats all skeleton-only baselines, and the 0.73% gap from multi-modal alignment may be noise.","rationale":"The reader's weakest assumption focuses on FoI target corruption, which is a plausible failure mode. However, the more load-bearing issue is statistical: the paper's own ablations show the entire multi-modal alignment contribution is only 0.73% on the main benchmark, and the skeleton encoder alone already surpasses all baselines. This means the central claim 'visual-informed benefits while preserving skeleton-only efficiency' could be true or false; the reported evidence does not separate the alignment effect from architectural gains or from noise. I agree with the reader's overall CONDITIONAL verdict because the concerns are addressable, but I do not fully agree that FoI reliability is the weakest point—the small and unvalidated alignment gain is more fundamental. The proposed test (multiple seeds + significance) directly settles whether the alignment effect is real. I recommend UNCHANGED because the reader's CONDITIONAL verdict already reflects the need for stronger evidence.","tokens_in":22940,"tokens_out":3453,"duration_ms":39104,"concrete_test":"Run the full STAR and the 'w/o Multi-modal Alignment' (or 'w/o FoI') configurations for at least 5 independent seeds on NTU Mutual 26 X-Sub and, ideally, on Chico/HARPER. Report mean±std and perform a paired significance test (e.g., bootstrap or Wilcoxon signed-rank) on the accuracy difference. If the 0.73% gap is not consistently reproduced or is not significant, the central visual-alignment claim is unsupported. Also report whether the skeleton-only encoder (No Alignment) beats all Table II baselines in every seed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that training-time alignment with visual RoI features materially improves skeleton-only inference. The paper's own ablation on NTU Mutual 26 X-Sub (Table III(b)) shows removing multi-modal alignment drops accuracy from 92.23 to 91.50 (Δ=0.73), and removing FoI drops to 91.48 (Δ=0.75). Yet the same table reports 'No Alignment' at 91.50, which already exceeds every skeleton-only baseline in Table II (best CHASE+CTR-GCN 91.30, ISTA-Net 90.56, MMCL 91.42). Thus the reported SOTA performance is largely attributable to the new skeleton encoder (ISTs/ER) rather than to the visual alignment. The 0.73% improvement is small, no error bars or multiple-seed variance are provided, and no significance test is reported; it could be within run-to-run variation. The FoI failure cases (Fig. 8) are secondary: even when the detector works correctly, the measured alignment benefit is small. Without statistical evidence that the alignment gap is real and robust across seeds, the paper's central claim that visual cues provide a meaningful benefit at skeleton-only inference is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STAR, a framework for skeleton-based human-robot and human-human interaction recognition. STAR consists of a skeleton encoder built on Entity Rearrangement (ER) and Interactive Spatiotemporal Tokens (ISTs), a visual branch that extracts interaction-focused RGB features via Focus on Interactions (FoI), and a contrastive alignment loss that trains skeleton and visual features in a shared latent space. At inference, only the skeleton branch is used, with an auxiliary refinement head that estimates visual-informed logits from skeleton features. The method is evaluated on Chico, HARPER, NTU Mutual 11, and NTU Mutual 26, reporting state-of-the-art accuracy. The paper also provides ablations for each component, robustness tests under noise and masking, and a public code release.","tokens_in":23334,"tokens_out":5669,"duration_ms":61063,"significance":"If the central claim holds, STAR would be a useful contribution: it demonstrates that visual cues can be distilled into a skeleton-only model through training-time alignment, preserving the efficiency and privacy advantages of skeleton-based inference while improving accuracy. The paper's strengths are its comprehensive evaluation across four benchmarks, including two recent HRI datasets, its component-wise ablations, and the release of code. The skeleton encoder design (ISTs and ER) appears well motivated and the authors provide quantitative and qualitative evidence that the learned representations are more discriminative. However, the paper's headline contribution—the multi-modal alignment—is supported by a small accuracy gain (about 0.7 percentage points on NTU Mutual 26) and lacks statistical validation. The alignment loss itself, as written, also raises a technical concern about whether it actually pulls positive pairs together. These issues must be addressed before the central claim can be considered established.","major_comments":[{"comment":"The claim that STAR is 'the first to leverage multi-modal alignment to learn both human-robot and human-human interactions' is stated in the introduction and related work, but the comparison to prior multi-modal alignment works (e.g., GAP, MMCL, C2VL) is made at the level of task scope. The paper does not discuss whether any of those methods could be adapted to the interaction setting with minimal changes. A more careful positioning, perhaps with an adapted-baseline experiment, would strengthen the novelty claim.","section":"§II-B"}],"minor_comments":[{"comment":"The sentence 'with a refinement head further refines predictions' has a grammatical error ('head further refines'). Please revise.","section":"Abstract / Intro"},{"comment":"The figure caption for Fig. 1 uses 'Ours can infer the interactions in dark or privacy-sensitive workspaces' but the figure shows only an illustrative example; consider clarifying that this is a schematic and not an actual experimental result.","section":"§I, Fig. 1"},{"comment":"The use of Chen et al. [107] to argue variance reduction from random permutation is appropriate, but the notation O = d πO and the approximate-invariance case are introduced briefly. A short intuitive explanation of why approximate invariance also yields variance reduction would improve readability.","section":"§III-A"},{"comment":"In Eq. (8), the operator ∩ is defined as 'intersection' with the original video, but the actual operation is a spatial crop using the maximum covering box. The notation is confusing; please use a clearer operator name, e.g., 'crop'.","section":"§III-C"},{"comment":"The table title 'PRETRAINED VISION ENCODERS' includes 'No Alignment' as a row, which is not a vision encoder. Consider moving that row to Table III(b) or renaming the table.","section":"§IV-E, Table III(a)"},{"comment":"The parameter counts in Table III(e) are given for different encoder layers, but the 'No Alignment' row reports a different parameter count (6.22M) that is not directly comparable to the 6.63M used for the default model. Please clarify whether the parameter count includes the alignment MLP and refinement head, and note that removing alignment reduces parameters.","section":"§IV-E, Table III(e)"},{"comment":"The robustness experiment applies noise with σ=0.01 and masking with p=0.01, but the choice of these values is not justified. A small sensitivity analysis over noise/mask levels would make the robustness claim more convincing.","section":"§IV-E, Table III(h)"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the code release is commendable. The main concern is that the paper's headline contribution—the multi-modal alignment—is supported by small accuracy differences without statistical grounding. The Eq. (9) sign issue is potentially serious and should be checked against the released code; if the code implements standard InfoNCE, the equation must be corrected. The authors may also want to include the HRI datasets in the multi-modal baseline comparison. I would ask for a revised version with multi-seed experiments, a corrected alignment loss description, and a quantitative analysis of detector failures before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Roughly: this is a solid, incremental extension of the authors' own ISTA-Net line. The new pieces are a Focus-on-Interactions RoI extractor for RGB, a contrastive alignment loss between skeleton and visual features, and a refinement head used at test time. The experiments are clean: four benchmarks, per-component ablations, robustness checks, and public code. The skeleton encoder alone already beats prior skeleton-only methods by a small margin, and the alignment adds another 0.5–0.8% on NTU Mutual 26. That is the paper's real situation: the gains are modest but consistent across ablations, and the framework is coherent.\n\nThe soft spots are about evidence strength, not logic. The central claim—that visual alignment helps skeleton-only inference—is supported by a single ablation number (0.73% on NTU 26 X-Sub) with no repeat runs, error bars, or significance tests. The stress-test note is right to press on this: 0.73% could be noise, and the paper doesn't demonstrate otherwise. The authors also tuned λ1, λ2, β, window shapes, and alignment layer per benchmark, which raises the usual overfitting-to-test-set concern. On Chico and HARPER there is no multi-modal baseline at all, so we can't see what the ceiling is for using RGB at inference. Those are addressable issues, but they're real.\n\nI'd send this to peer review. The application to HRI datasets is timely, the code release matters, and the skeleton encoder (ISTs + ER) is a legitimate contribution even if the alignment story ends up being smaller than advertised. For the revision, I'd ask for multiple seeds, mean±std, an ablation of the alignment loss on at least one HRI dataset, and ideally a simple RGB-at-inference baseline for context. I'd also ask them to soften the 'visual-informed benefits' framing if the numbers don't hold up.\n\nFor a colleague working on skeleton-based interaction recognition or HRI, this is worth reading and citing. For the rest, it's a competent, unflashy extension.","headline":"Competent extension of ISTA-Net with multi-modal alignment; the visual-alignment claim is plausible but only weakly evidenced, so the revision needs statistical rigor.","tokens_in":23782,"tokens_out":3752,"would_cite":true,"duration_ms":38715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a skeleton-only interaction recognizer can learn from RGB video during training—via contrastive alignment of skeleton and visual interaction features—and then outperform previous methods at inference using only skeleto","keywords":["Interaction recognition","Skeleton-based action recognition","Human-robot interaction","Multi-modal alignment","Contrastive learning","Spatiotemporal tokens","Entity permutation invariance","Visual distillation"],"falsifier":"Run STAR on a test set where the two interacting entities are frequently partially out of frame or far apart; if accuracy drops to the no-alignment baseline (about 91.5% in the paper's Table III) while the alignment loss stays high, the visual teacher is injecting noise rather than signal, and random-crop controls would confirm whether FoI localization is the source of gains.","tokens_in":22879,"feed_emoji":"🦴","tokens_out":5851,"duration_ms":60056,"temperature":0.7,"pith_summary":"STAR tackles two problems in skeleton-based interaction recognition: skeletons of two interacting entities carry less evidence than video, and standard skeleton architectures often fail to model the interaction itself. The proposed solution trains with both modalities but deploys with only skeletons. A skeleton encoder built from Interactive Spatiotemporal Tokens and Entity Rearrangement learns interaction-local features while respecting that the two actors could be swapped. During training, a contrastive loss pulls those skeleton features toward visual features cropped around the interaction in the RGB video; at test time only skeletons are used, with an optional refinement head that learned to mimic the visual branch. If the claim holds, a privacy-preserving, low-light-friendly skeleton system gains the disambiguating power of video—distinguishing, for example, 'punch/slap' from 'hit with object'—without needing video at runtime.","feed_headline":"Video-trained skeletons top four interaction-recognition benchmarks","feed_subtitle":"RGB video is used only during training, so low-light and privacy-sensitive deployments still get visual disambiguation.","key_machinery":"Two mechanisms carry the argument. Entity Rearrangement randomly permutes the order of the two entities during training; because interaction labels are invariant under entity swap, the symmetric group reduces variance in the estimator and stabilizes optimization. Interactive Spatiotemporal Tokens are 3D windows sliding over time, joints, and entities, so each token bundles a local spatiotemporal patch of the interaction; stacked Token Self-Attention blocks then model interdependencies without relying on adjacency matrices, which matters because human bone structure differs from a quadruped robot's. The third mechanism is the multimodal alignment: a contrastive loss ties the skeleton token fe","core_discovery":"STAR is claimed to be the first skeleton-based interaction recognition method to use multi-modal alignment with visual interaction features. In training, an object detector finds both entities in each RGB frame; a maximum square box covering all detections at all sampled times crops an interaction Region of Interest, which a pretrained video encoder embeds. The skeleton encoder's intermediate feature vector is aligned to that visual embedding by a contrastive objective in a shared latent space, and a separate refinement head learns to predict the label from the visual embedding, so at test time it can be fed with the skeleton feature instead. On the Chico and HARPER human-robot datasets and","pith_inferences":["A natural extension of the training-time-video, inference-time-skeleton recipe is to distill knowledge from larger video foundation models without increasing deployment cost; the paper's encoder benchmark suggests video models transfer more useful cues than image models.","The known-actor-type assumption flagged in the conclusion could be tested by clustering roles from spatiotemporal movement patterns; if that works, STAR-style alignment could apply to open-world human-robot interaction instead of pre-specified entity pairs.","The FoI failure cases suggest a specific stress test: when the detector misses entities or crops too large a region, the contrastive objective may pull skeleton features toward noise; probing performance under progressively larger entity distances would reveal how much robustness margin remains.","Because the visual branch is a training-time teacher only, the same alignment objective could be applied to other privacy-sensitive modalities (e.g., depth or thermal) to enrich skeleton features without changing the skeleton-only inference pipeline."],"forward_implications":["If STAR is correct, a skeleton-only model can learn from video during training and match or beat methods that need video at inference, enabling privacy-sensitive and low-light deployments.","The Entity Rearrangement perspective implies that two-entity interaction modeling can treat entity order as a symmetry, reducing reliance on subject-specific adjacency priors in graph-based skeleton models.","Because the visual target comes from Focus on Interactions cropping, alignment quality depends on detector localization; the paper shows success cases under occlusion and identifies partially out-of-frame or far-apart entities as failure cases.","The refinement head adds a 'think-twice' step at test time with negligible overhead (about 0.41M additional parameters reported for the alignment components).","On fine-grained categories such as 'hit with object' and 'punch/slap', visual alignment yields large reported category-level accuracy gains, suggesting video cues specifically resolve ambiguities in contact point and manipulated object."],"fun_headline_variants":["Skeleton-only inference, RGB-trained: STAR tops interaction benchmarks","Video-trained skeleton features win on four interaction datasets","STAR blends skeleton and RGB cues for top interaction recognition","Visual cues train skeleton model that runs without RGB at test"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the Focus on Interactions crops—obtained from a pretrained object detector and a maximum square box—contain the interaction-discriminative visual information that survives temporal sampling, and that the detector does not fail; the paper itself notes (Fig. 8) failures when entities are partially out of frame or too far apart, and the conclusion acknowledges the assumption of known actor types.","fun_headline_variants_meta":{"raw":{"variants":["Skeleton-only inference, RGB-trained: STAR tops interaction benchmarks","Video-trained skeleton features win on four interaction datasets","STAR blends skeleton and RGB cues for top interaction recognition","Visual cues train skeleton model that runs without RGB at test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2251,"prompt_tokens":795,"completion_tokens":1456,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1390}},"tokens_in":539,"tokens_out":1456,"duration_ms":11925,"temperature":1.0,"reasoning_tokens":1390,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:15:59.012018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run STAR on a test set where the two interacting entities are frequently partially out of frame or far apart; if accuracy drops to the no-alignment baseline (about 91.5% in the paper's Table III) while the alignment loss stays high, the visual teacher is injecting noise rather than signal, and random-crop controls would confirm whether FoI localization is the source of gains.","supporting_citations":[],"review_version":1}