{"id":"68b6459f-c4a8-4a35-83fb-f8891808cc88","arxiv_id":"2508.04513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An autoencoder with joint-disentangled embeddings and vector-quantized temporal patches segments skeleton sequences into actions without labels, beating prior unsupervised methods on HuGaDB, LARa, and two of three BABEL subsets.","lead":"This paper introduces Skeleton Motion Quantization (SMQ), an unsupervised method that splits joint-level skeleton embeddings into one-second patches and groups them into discrete motion words to segment long action sequences. The authors report improvements over prior unsupervised methods on three skeleton datasets, with no labeled actions needed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Motion words may not be action-consistent: the one-to-one Hungarian mapping (suppl. 6.2) assumes one code per action, but Table 16 shows LARa MoF is higher at K=4 (44.0) than at the true K=8 (37.4), suggesting larger codebooks fragment actions rather than capturing action identity.","rationale":"The reader's weakest_assumption is in the same general area: the codebook assignment is assumed to produce action-consistent labels. I agree partially; I do not think temporal continuity is the core issue, since patch size and EMA already provide smoothing. The sharper, falsifiable issue is the interaction between the one-to-one Hungarian evaluation and the learned codebook, and the paper's own K-sweep in Table 16 provides internal evidence of fragmentation. This is more specific than the reader's 'no analysis of what motion words encode' and does not require new external data. The concern is load-bearing because the method's semantic mechanism is central to the paper's contribution, yet the SOTA comparison could still hold under the same protocol even if the mechanism is different. I recommend keeping the reader's CONDITIONAL verdict: the issue warrants additional analysis but does not, on its own, overturn the empirical comparison. The proposed contingency-table/oracle-merge test would settle whether the learned motion words are actually action-consistent or merely reconstruction-friendly pose clusters.","tokens_in":19693,"tokens_out":7300,"duration_ms":94471,"concrete_test":"With the released code, train SMQ on LARa with K=8 and build the 8x8 contingency table between codebook indices and ground-truth action labels at the patch level. Compute, for each GT action, the entropy of codebook assignment. Then apply a post-hoc oracle merge: hierarchically merge codebook indices with similar GT distributions until K clusters remain, and recompute MoF and F1@50 with this merged labeling (still using global Hungarian on the merged clusters). If per-action code entropy is high, or if merged F1@50 exceeds the reported 16.4 by a nontrivial margin (e.g., >5 points), the one-to-one mapping is artificially capping performance and the learned motion words are not action-consistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SMQ's skeleton motion words drive semantically meaningful action clustering rests on the assumption that nearest-neighbor codebook assignment (Eq. 1) yields labels aligned with actions. That assumption is load-bearing because the evaluation protocol (suppl. 6.2) applies global Hungarian matching with a one-to-one mapping between the K codebook indices and the K ground-truth action classes. The model is never trained to make this mapping one-to-one; Eqs. 5-7 only demand reconstruction and encoder commitment. The paper's own K-robustness study (Table 16, LARa) is internal evidence that the assumption fails: with the true K=8, MoF is 37.4, while K=4 gives 44.0 and K=5 gives 41.9. If each action corresponded to one learned motion word, providing the true number of actions should not be clearly worse than providing half the number of actions. The most plausible explanation is that the 8-code codebook assigns multiple codes to sub-patterns within actions (pose/gait phases), and the one-to-one Hungarian match then forces those codes onto other actions, fragmenting segments and capping F1. The paper supplies only t-SNE visualizations as evidence of what motion words encode, and no per-code/ground-truth contingency analysis. Hence the reported SOTA gains may reflect reconstruction-driven pose clustering plus the Hungarian one-to-one constraint rather than action-consistent motion words. This does not by itself overturn the empirical comparison, but it makes the stated mechanism questionable and the absolute scores somewhat arbitrary with respect to K.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SMQ, an unsupervised method for skeleton-based temporal action segmentation. A sequence-to-sequence temporal autoencoder processes each joint independently to obtain a disentangled latent embedding; the latent sequence is split into non-overlapping patches, each patch is quantized to the nearest of K learnable codebook vectors ('skeleton motion words'), and the decoder reconstructs the input from the quantized patches. Frame-level segmentation is read out directly from the nearest-codebook assignments. The method is evaluated on HuGaDB, LARa, BABEL (three subsets), and PKU-MMD v2, and is reported to outperform prior unsupervised temporal action segmentation methods and self-supervised skeleton representation baselines on most metrics.","tokens_in":20125,"tokens_out":3932,"duration_ms":46721,"significance":"If the empirical claims hold, SMQ would be a meaningful advance for unsupervised skeleton-based temporal action segmentation, a setting where prior unsupervised video-segmentation methods transfer poorly. The paper provides a clear architectural contribution—joint-disentangled embeddings plus patch-level vector quantization—and backs it with a broad set of ablations, qualitative results, runtime measurements, and public code. The main numerical gains on HuGaDB and LARa are large, and the extension to the BABEL protocol is a useful stress test. However, the central interpretive claim that the learned codebook entries are action-consistent 'motion words' is not established; in fact, the paper's own K-robustness experiment provides evidence against a one-code-per-action mapping. The evaluation also has methodological weaknesses—hyperparameter selection on the reported test set, missing error bars, and a modified BABEL protocol—that must be clarified before the comparison can be fully trusted.","major_comments":[{"comment":"The codebook size K is set to the number of ground-truth action classes (Suppl. 6.1), and evaluation aligns clusters to labels via a global one-to-one Hungarian matching (Suppl. 6.2). The K-robustness study (Table 16) shows that on LARa, K=4 achieves MoF 44.0 while the true K=8 achieves only 37.4. If each learned motion word corresponded to an action, providing the true number of actions should not be clearly worse than providing half that number. This is internal evidence that the learned codebook entries fragment action classes and that the one-to-one Hungarian mapping then distorts the evaluation. The paper's central claim that skeleton motion words are semantically meaningful action clusters is therefore not supported by the current evidence. Please report a codebook-to-ground-truth contingency analysis (e.g., per-code action purity or a many-to-one mapping), and discuss why the true","section":"Table 16 and Suppl. 6.1–6.2"},{"comment":"The patch size, reconstruction loss weight λ, and EMA decay α are selected on LARa by comparing test-set metrics, and the final LARa results are then reported as the method's performance. This is selection on the test set and can inflate the reported numbers, especially in an unsupervised setting where no validation split is described. The same selected hyperparameters are then fixed across the other datasets, so the issue propagates. Please either use a validation split for hyperparameter selection, report all configurations' performance across datasets, or at minimum disclose this dependence and supplement the main tables with multiple-seed means and standard deviations. Given the absence of error bars, it is currently impossible to tell whether the reported gains are within run-to-run variation.","section":"Sec. 4.3, Tables 5, 7, 8"},{"comment":"The BABEL evaluation modifies the protocol of [69] by excluding sequences containing more than 50% background actions. This is a significant alteration: it changes the difficulty and composition of the benchmark, and it affects the comparability of the reported numbers with any previously published numbers. The paper needs to state how many sequences are removed from each subset, report results on the original unmodified subset as well, and confirm explicitly that every baseline method was run on exactly the same filtered sequences. Without this, the BABEL comparison in Table 2 is not fully controlled.","section":"Sec. 4.1 (BABEL protocol)"},{"comment":"No variance or significance information is reported for the main results. Unsupervised clustering methods, including SMQ, depend on initialization and stochastic training; the differences between SMQ and the strongest baseline on some BABEL metrics are small (e.g., ASOT vs SMQ on Subset-1 Edit and F1@10 in Table 2). Reporting single runs is insufficient to support the claim of consistent superiority. Please report at least three seeds with means and standard deviations for all principal comparisons, and indicate whether the differences are stable across seeds.","section":"Tables 1–3 and 16"}],"minor_comments":[{"comment":"The summation notation in the commitment loss is garbled ('Σ_{i∈ p}'); it should read as a sum over all patches in the batch.","section":"Eq. (6)"},{"comment":"The sentence 'The codebook size corresponds to the number of ground-truth actions in the dataset as it is required by the protocol' is a key design choice and should be stated prominently in the main text, not only in the supplement. It also deserves a justification beyond 'required by the protocol.'","section":"Suppl. 6.1"},{"comment":"The column header 'Num of actions (K)' conflates the codebook size with the number of ground-truth actions. Since the two are set equal by the protocol, this is understandable, but a clearer caption would avoid confusion about what is being varied.","section":"Table 16 caption"},{"comment":"The text says 'ASOT performs slightly better for the other metrics' for BABEL Subset-1, but the table shows ASOT also outperforms SMQ on Edit and all F1 thresholds there. The wording understates the comparison; please be precise.","section":"Sec. 4.2, text after Table 2"},{"comment":"The qualitative figure would benefit from clearer axis labels and legends indicating which color corresponds to which action; the current figure is hard to read without the caption.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The K-robustness result (Table 16) is the key risk. It challenges the paper's central interpretability claim and could indicate that the reported gains come from reconstruction-driven pose clustering plus a favorable Hungarian alignment rather than from action-consistent motion words. The empirical comparison may still be salvageable, but the authors need to provide a direct analysis of what the learned codebook entries encode and how they map to ground-truth actions. If that analysis is not convincing, the paper's contribution would need to be reframed substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is the first paper I know of that does unsupervised temporal action segmentation directly on long skeleton sequences, and it works, in the sense that it beats the video-based baselines (CTE, TOT, ASOT) when those are run on skeleton input. The combination of joint-disentangled TCN embeddings, temporal patching, and VQ codebooks is sensible, and the ablations actually support those choices. Code is public. If you work on skeleton activity understanding, you should know about it.\n\nNow the soft spots, in rough order of how much they bother me.\n\nThe K-robustness table (Table 16) is the one that bit me. The protocol sets K to the true number of actions, which is standard, but their own sweep on LARa shows MoF 44.0 at K=4 vs 37.4 at K=8. That is direct evidence that the codebook does not learn one motion word per action. The one-to-one Hungarian mapping used for evaluation then forces a correspondence that the model never learned. The paper's 'semantically meaningful action clusters' claim is therefore unsupported; the reported numbers are real under the protocol, but the interpretation is speculative. They need per-code/ground-truth contingency tables, not just t-SNE.\n\nThe evaluation has avoidable leaks:\n- Hyperparameters (patch size, lambda, EMA decay) are selected on LARa and then reported on LARa. That's test-set tuning. Held-out validation or at least an honest description is needed.\n- No error bars anywhere. Given the random init and EMA updates, I'd expect variance.\n- The BABEL protocol modifies [69] by excluding sequences with >50% background. That may be defensible, but it needs a sensitivity analysis or a comparison on the unmodified subset.\n- HVQ is cited but never compared. Since it's the closest VQ-based method, the absence is conspicuous.\n\nThe 'large margin' in the abstract is also not true everywhere: on BABEL Subset-1, ASOT beats SMQ on Edit and F1.\n\nNone of this kills the empirical contribution. The comparison to baselines is fair (all run on the same inputs and the same Hungarian protocol), and the improvements on HuGaDB and LARa are large. It's just that the mechanism is in question and the evaluation needs tightening.\n\nMy recommendation: send it to peer review, but expect major revision. The authors can probably fix the tuning and error bars in a week, and add HVQ. The K issue needs a real experiment—e.g., silence on K or a learned K from silhouette—and a direct discussion of what the codebook entries actually encode.\n\nWorth a reading group slot, too.","headline":"First unsupervised skeleton-based TAS method; solid empirical study, but the motion-word story is fragile and the evaluation has several fixable leaks.","tokens_in":20593,"tokens_out":3316,"would_cite":true,"duration_ms":35062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMQ shows that unsupervised skeleton-based temporal action segmentation can be done by learning a small codebook of discrete skeleton motion words, and that joint-disentangled, patch-level quantization outperforms prior unsupervised methods","keywords":["unsupervised temporal action segmentation","skeleton-based action segmentation","vector quantization","motion words","temporal autoencoder","joint-disentangled embedding","codebook learning","human motion analysis"],"falsifier":"Concrete test: take a trained SMQ model on LARa and compute, for each ground-truth action, the entropy of the codebook-word distribution over its patches. If a single repeated action instance is split across many words (high intra-action entropy), or if the same word appears across actions with dissimilar poses, the action-consistency claim fails. Alternatively, evaluate SMQ on a dataset with two actions that share poses but differ in temporal order: if quantization groups by pose rather than action, the two actions will collapse into one segment.","tokens_in":19609,"feed_emoji":"🦴","tokens_out":8155,"duration_ms":86097,"temperature":0.7,"pith_summary":"Unsupervised temporal action segmentation has been studied mostly on RGB video; skeleton recordings—robust, privacy-preserving, and common in wearables and motion capture—have lacked a working unsupervised recipe. This paper tries to close that gap by recasting the task as learning a small vocabulary of discrete skeleton motion words: a temporal autoencoder maps each joint separately into a latent space, cuts the latent sequence into one-second patches, and quantizes every patch to its nearest codebook vector. The codebook index becomes the frame's action label, so segmentation and clustering happen in a single pass with no labels. Across HuGaDB, LARa, and BABEL subsets, the method reports consistent gains over prior unsupervised video-action-segmentation methods and over self-supervised skeleton representation baselines, with the largest margins on segment-level metrics.","feed_headline":"Quantized motion words label skeleton actions without supervision","feed_subtitle":"One-second motion-word patches outperform transferred video methods on three skeleton datasets.","key_machinery":"The central object is the skeleton motion word: a codebook vector $c_k \\in \\mathbb{R}^{P \\times (V \\cdot D)}$ representing a prototypical short skeleton motion over a $P$-frame patch. It carries the argument through four moves: (1) the encoder processes each joint independently so the embedding space stays joint-disentangled; (2) Euclidean nearest-neighbor assignment converts each latent patch to a word and yields the segmentation $Y_{nt}$ directly from the assigned word index; (3) EMA codebook updates and a stop-gradient commitment loss stabilize the learned vocabulary; and (4) a translation/rotation-invariant inter-joint distance reconstruction loss shapes the words to preserve pose-level","core_discovery":"SMQ treats unsupervised skeleton-based temporal action segmentation as online time-series clustering: a dilated-TCN sequence-to-sequence autoencoder maps each joint separately into a latent space, the per-joint embeddings are patched into non-overlapping one-second segments, and each patch is assigned to the nearest vector in a learned codebook whose entries are the \"skeleton motion words.\" The assigned codebook index directly yields the frame-level action label, and training with an inter-joint-distance reconstruction loss plus a commitment loss makes the words capture actions rather than pose noise. On HuGaDB, LARa, and the three BABEL subsets, SMQ reports higher MoF, edit score, and segme","pith_inferences":["One step the paper leaves implicit: the same codebook could feed a hierarchical scheme—coarse words for actions, finer words for sub-motion—which would directly address the boundary detection the paper names as future work.","The supplementary silhouette-score analysis suggests $K$ (the number of motion words) can be chosen from unlabeled data; testing that heuristic on HuGaDB and BABEL, rather than only LARa, is a concrete next check.","If motion words are shared across actions, they may function as reusable movement primitives; inspecting per-word mean poses and joint-velocity profiles could test whether the vocabulary composes actions from smaller parts.","The graceful degradation under missing joints points beyond benchmarks: in deployments with partial occlusion or dropped IMU nodes, SMQ might keep segmenting with only a small score drop—a property worth verifying on other datasets."],"forward_implications":["Because labels are the codebook indices themselves, SMQ needs no separate clustering step or Viterbi-style post-processing, avoiding the over-fragmentation of CTE/TOT and the over-merging of ASOT.","Patch-level quantization matters: frame-wise assignment over-segments, while one-second patches give the best segmental metrics on LARa; patch size is reported as a key temporal-resolution knob.","Keeping joints disentangled in the embedding improves performance on all three datasets, so skeleton structure itself carries signal that joint-concatenated representations lose.","Reconstruction through inter-joint distances outperforms per-coordinate MSE, indicating that a pose-level, translation- and rotation-invariant objective is part of what makes the learned motion words useful.","SMQ degrades gracefully when joints are randomly dropped (25% and 50% missing), indicating the learned motion words are not tied to any single joint."],"supporting_citations":[{"why":"Supplies the vector-quantization/EMA codebook machinery that SMQ adapts to skeleton motion patches.","marker":"[60]"},{"why":"CTE is the state-of-the-art unsupervised video-action-segmentation baseline SMQ outperforms on HuGaDB.","marker":"[26]"},{"why":"TOT provides the joint representation-learning/online-clustering baseline compared across all datasets.","marker":"[27]"},{"why":"ASOT is the strongest prior method on LARa and the BABEL subsets; SMQ's gains are measured against it.","marker":"[64]"},{"why":"HuGaDB supplies the lower-limb IMU gait/activity sequences used for the first main evaluation.","marker":"[9]"},{"why":"LARa supplies the warehouse mocap skeleton sequences and the 50 fps protocol used for the second main evaluation.","marker":"[45]"},{"why":"BABEL supplies the full-body mocap sequences from which the three four-action subsets are built.","marker":"[47]"},{"why":"MS-TCN is the dilated temporal convolutional architecture the SMQ autoencoder is built on.","marker":"[12]"},{"why":"MS-GCN sets the supervised performance reference and supplies the LARa preprocessing used for the skeleton input.","marker":"[14]"},{"why":"Predict&Cluster is one of the self-supervised skeleton representation baselines evaluated and outperformed in Table 3.","marker":"[54]"}],"fun_headline_variants":["Skeleton motion words map actions without any labels","Motion-word patches segment skeleton actions unsupervised","Unsupervised skeleton segmentation via motion-word codebook","One-second skeleton patches become action words, no labels","Skeleton autoencoder yields motion words for action labeling"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The method stands or falls on whether assigning each motion patch to the closest learned codebook word, guided only by reconstruction and commitment losses, yields labels that match actions; if the codebook organizes by pose details instead of action identity, the segmentation fragments and the segmental metrics collapse.","fun_headline_variants_meta":{"raw":{"variants":["Skeleton motion words map actions without any labels","Motion-word patches segment skeleton actions unsupervised","Unsupervised skeleton segmentation via motion-word codebook","One-second skeleton patches become action words, no labels","Skeleton autoencoder yields motion words for action labeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2674,"prompt_tokens":689,"completion_tokens":1985,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":1913}},"tokens_in":433,"tokens_out":1985,"duration_ms":17030,"temperature":1.0,"reasoning_tokens":1913,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:55:43.469038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concrete test: take a trained SMQ model on LARa and compute, for each ground-truth action, the entropy of the codebook-word distribution over its patches. If a single repeated action instance is split across many words (high intra-action entropy), or if the same word appears across actions with dissimilar poses, the action-consistency claim fails. Alternatively, evaluate SMQ on a dataset with two actions that share poses but differ in temporal order: if quantization groups by pose rather than action, the two actions will collapse into one segment.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the vector-quantization/EMA codebook machinery that SMQ adapts to skeleton motion patches."},{"cited_title":"Unsupervised learning of action classes with continu- ous temporal embedding","cited_arxiv_id":null,"evidence_quote":"CTE is the state-of-the-art unsupervised video-action-segmentation baseline SMQ outperforms on HuGaDB."},{"cited_title":"Unsupervised action segmentation by joint representation learning and on- line clustering","cited_arxiv_id":null,"evidence_quote":"TOT provides the joint representation-learning/online-clustering baseline compared across all datasets."},{"cited_title":"Temporally consistent unbal- anced optimal transport for unsupervised action segmenta- tion","cited_arxiv_id":null,"evidence_quote":"ASOT is the strongest prior method on LARa and the BABEL subsets; SMQ's gains are measured against it."},{"cited_title":"Hugadb: Hu- man gait database for activity recognition from wearable in- ertial sensor networks","cited_arxiv_id":null,"evidence_quote":"HuGaDB supplies the lower-limb IMU gait/activity sequences used for the first main evaluation."},{"cited_title":"Lara: Creating a dataset for human activity recognition in logistics using semantic attributes","cited_arxiv_id":null,"evidence_quote":"LARa supplies the warehouse mocap skeleton sequences and the 50 fps protocol used for the second main evaluation."},{"cited_title":"Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J","cited_arxiv_id":null,"evidence_quote":"BABEL supplies the full-body mocap sequences from which the three four-action subsets are built."},{"cited_title":"Ms-tcn: Multi-stage temporal convolutional network for action segmentation","cited_arxiv_id":null,"evidence_quote":"MS-TCN is the dilated temporal convolutional architecture the SMQ autoencoder is built on."},{"cited_title":"Skeleton-based action segmentation with multi-stage spatial- temporal graph convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"MS-GCN sets the supervised performance reference and supplies the LARa preprocessing used for the skeleton input."},{"cited_title":"Predict & cluster: Unsupervised skeleton based action recognition","cited_arxiv_id":null,"evidence_quote":"Predict&Cluster is one of the self-supervised skeleton representation baselines evaluated and outperformed in Table 3."}],"review_version":1}