{"id":"9083b8bf-0d7c-4dd7-8e46-a2f1fc34d638","arxiv_id":"2502.05432","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MoFM pretrains a vision transformer on discrete tokens of human motion heatmaps and fine-tunes it for action recognition and anomaly detection, but the gains from pretraining are not isolated.","lead":"MoFM is a pretrained neural network that turns human skeletons into a compact 'motion vocabulary' and then learns general patterns from that vocabulary. It is tested on four tasks, action recognition, one-shot recognition, and two anomaly detection settings, with results near but below current best models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No from-scratch baseline and no pretraining data-scale disclosure; the transfer evidence is uncontrolled and cannot support the foundation-model claim.","rationale":"The reader's weakest assumption about max-pooling over joints is technically sound: the dVED decoder reconstructs only the pooled heatmap UR (Sec. 4.2, Eq. 5), so the discrete tokens are not required to preserve joint identity, and a simple linear head cannot recover information that the pretraining target discarded. I partially agree with this. However, the more decisive problem is the uncontrolled comparison: because downstream experiments fine-tune the entire backbone and head, the model can compensate for a weak pretraining objective during task-specific training. The absence of any random-initialization baseline means the observed accuracies cannot be attributed to pretraining at all. The missing pretraining data-scale disclosure and the non-standard one-shot protocol independently undermine the 'large-scale foundation model' narrative. These issues strengthen, rather than change, the reader's rejection of the paper in its current form.","tokens_in":19118,"tokens_out":5812,"duration_ms":65918,"concrete_test":"Run the exact same MoFM architecture (dVED tokenizer plus ViT backbone plus FC head) from random initialization, with the same downstream hyperparameters in Tables 6-9, on NTU-60 X-Sub, SHT, and UBnormal. If the randomly initialized model matches or beats MoFM-FC within 1-2 points, then pretraining contributes no transferable representation and the foundation-model claim is unsupported. As a secondary check, freeze the MoFM backbone and train only the head; if a randomly initialized frozen backbone performs equally, the result is even more decisive. Also report the exact pretraining dataset size in videos, frames, and hours.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MoFM's pretrained backbone transfers to diverse downstream tasks. The reported experiments do not establish this because there is no comparison against an identically architected backbone trained from scratch. All four downstream protocols fine-tune the full pretrained model plus a head (supplementary Tables 6-9 give epochs and learning rates for the whole network), so any observed accuracy could come from task-specific fine-tuning alone, not from the pretrained representation. The numbers are also consistently below state of the art: NTU-60 X-Sub 89.4 vs 93.1, NTU-120 X-Sub 78.0 vs 88.9, one-shot NTU-120 56.6 vs 57.6, SHT self-supervised AUC 76.56 vs 82.90, and UBnormal 69.55 vs 78.1. Additionally, the pretraining corpus size is never quantified: the dVED is trained on CMU Panoptic (Sec. 4.2), but the MoFM backbone pretraining dataset, number of videos/frames, and token-level masking budget are absent, making the 'large-scale' claim untestable. The DT2 one-shot protocol is also non-standard: it uses a 100-sample auxiliary labeled set for supervised contrastive training and evaluates with 20 exemplars per remaining class, which is not the conventional one-shot evaluation. The claimed foundation behavior therefore rests on an uncontrolled transfer analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MoFM proposes a motion foundation model pipeline: 2D skeleton sequences are converted into Gaussian spatio-temporal heatmaps, divided into Thermal Cubes, discretized into MotionBook tokens by a discrete variational encoder-decoder (dVED), and used for BERT-style masked self-supervised training of a vision transformer backbone. The pretrained backbone is then fine-tuned with simple fully connected heads on four downstream tasks: NTU-60/NTU-120 action recognition, NTU-120 one-shot action recognition, self-supervised anomaly detection on ShanghaiTech and HR-ShanghaiTech, and supervised anomaly detection on UBnormal. The paper claims that the resulting model is a task-agnostic foundation backbone supporting one-shot, unsupervised, and supervised paradigms. The supporting evidence consists mainly of downstream fine-tuning results; no from-scratch baseline, no pretraining-scale disclosure, and no ablations of the proposed masking or tokenization components are reported.","tokens_in":19393,"tokens_out":6788,"duration_ms":66273,"significance":"The intended contribution is potentially useful: a discrete motion vocabulary plus a shared backbone that transfers across several skeleton-based tasks would be a valuable resource for the community. The mathematical components (Gaussian heatmaps, Gumbel-Softmax approximation, ELBO objective) are standard and are described correctly, and the supplementary tables provide enough hyperparameters to reproduce the fine-tuning protocols. The qualitative dVED reconstruction and the explicit discussion of the ghosting effect around moving joints are also positive aspects. However, as it stands, the experimental evidence does not support the central 'foundation model' claim, because the transfer results are uncontrolled and the scale of pretraining is never quantified.","major_comments":[{"comment":"The central claim is that the pretrained MoFM backbone transfers across diverse downstream tasks. All four downstream protocols fine-tune the full pretrained backbone plus a simple head, but the paper never compares against an identically architected backbone trained from scratch on each downstream task, nor against a randomly initialized backbone with the same head. Without such a control, the reported accuracies may be attributable to task-specific fine-tuning alone rather than to the pretrained representation. The claim is therefore not supported by the reported experiments. No error bars or significance testing are provided either, which makes the small differences against state-of-the-art methods even harder to interpret.","section":"Section 5, Tables 1-5; Supplementary Tables 6-9"},{"comment":"The paper repeatedly describes MoFM as 'large-scale' and 'trained on a large corpus of motion data,' but it never specifies the corpus used to pretrain the MoFM backbone. No dataset name, number of videos, number of frames, hours of motion, or number of training clips is given, and the token-level masking budget is also absent. The only pretraining data identified anywhere is CMU Panoptic, and that is described specifically for dVED training. If the backbone was also trained only on CMU Panoptic, the 'large-scale' claim is unsupported; if a different corpus was used, it must be disclosed. As written, the central scale claim is untestable.","section":"Sections 4.2-4.3"},{"comment":"The keypoint-wise masking scheme is presented as a contribution, but no quantitative ablation compares it with the blockwise masking of [7], with random cube masking, or with different masking budgets. The paper states that blockwise masking 'reduced the loss error' but produced qualitatively inaccurate reconstructions; these qualitative observations need to be backed by quantitative results on a downstream task. Similarly, there is no ablation of the vocabulary size, the dVED tokenizer versus continuous input features, or the number of pretraining epochs. Without these ablations, the contribution of the self-supervised pretraining objective to the downstream numbers is unclear.","section":"Section 4.3, Algorithm 1"},{"comment":"The one-shot action recognition protocol is non-standard. The model is trained on an auxiliary set of 100 labeled samples per class with supervised contrastive learning, and evaluation uses 20 exemplars for each remaining class. This is not the conventional one-shot setting, which typically provides a single labeled example per novel class or uses episode-based N-way K-shot evaluation. The comparison with prior methods in Table 3 may therefore not be apples-to-apples. The protocol should be justified or replaced with a standard episodic evaluation before the one-shot claim can be accepted.","section":"Section 5.2 and Table 3"},{"comment":"The dVED is trained to reconstruct only UR, which is obtained by max-pooling the joint-indexed heatmap U over the J joints, rather than reconstructing the full joint-aware tensor U. Because the discrete MotionBook tokens are learned under this joint-agnostic objective, the learned vocabulary may not preserve joint identity. This is potentially load-bearing for downstream tasks such as action recognition, where the identity of the joint matters. Please provide evidence that joint identity is preserved (for example, token prediction accuracy per joint, or an experiment with shuffled or permuted joint ordering), or change the dVED objective to reconstruct U.","section":"Section 4.1 and Eq. (5)"}],"minor_comments":[{"comment":"The text says the results are 'on par with or surpassing SOTA approaches,' but Tables 1 and 2 show MoFM-FC below the best prior methods on every split (for example, NTU-60 X-Sub 89.4 versus 93.1 for Duan et al., and NTU-120 X-Sub 78.0 versus 88.9 for Jang et al.). The wording should be aligned with the reported numbers.","section":"Section 5.1"},{"comment":"The entry for Zaheer et al. [110] reads '78-93', which appears to be a typo for a numeric value such as 78.93; please correct the formatting.","section":"Table 4"},{"comment":"The heading contains a typo, 'Calssification', which should be 'Classification'.","section":"Supplementary Section 7.2"},{"comment":"The notation for indices is confusing: the pose skeleton is written as P = {Pi}_{i=0}^{F-1} as if i indexes frames, while the heatmap equation uses i for a spatial coordinate, j for a joint, and f for a frame. Please clarify the index conventions for frames, joints, and spatial coordinates.","section":"Section 3 and Eq. (1)"},{"comment":"Several references are duplicated under different numbers (e.g., [17]/[18], [80]/[81], [85]/[86], [104]/[105]), and some in-text citations point to the wrong duplicate; the bibliography should be unified. In addition, Figure 4 uses the term 'PoseBook' at the bottom, whereas the body text consistently uses 'MotionBook'; the terminology should be made consistent.","section":"References and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The key uncertainty is whether a large pretraining corpus actually exists. The body of the paper mentions CMU Panoptic only for the dVED and never names the backbone pretraining data. If the MoFM backbone was also trained only on CMU Panoptic, the 'large-scale' foundation-model claim would be false, and the paper should be rejected. Please ask the authors to disclose the exact pretraining corpus, its size, and its curation process before a revision is considered. The missing from-scratch baseline and ablations are, in principle, addressable within a revision, but the scale disclosure is a factual prerequisite that cannot be substituted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely interesting piece of this paper is the keypoint-wise masking strategy for BERT-style pretraining on spatio-temporal heatmaps. That is a real adaptation to the sparsity of skeleton heatmaps, and the authors deserve credit for noticing that blockwise masking often lands on empty regions. The dVED tokenizer plus MotionBook is a reasonable pipeline, and the math (Gumbel-Softmax, ELBO) is standard and correctly described. The paper is also honest about not aiming to beat SOTA, which makes the below-SOTA numbers less damning than they would otherwise be.\n\nThe soft spot is exactly what the stress-test note says: there is no from-scratch baseline. Since all downstream fine-tuning updates the whole pretrained backbone, the reported accuracies could come from task-specific training alone. Without an identically architected backbone trained from scratch, or at least a linear-probing comparison, the transfer value of the pretrained representation is not established. The pretraining data scale is also never quantified—'large corpus' is a slogan, not a fact, and the MoFM backbone pretraining set (number of videos, frames, masking budget) is absent. The 'large-scale' in the title is untestable. I'd also flag a mismatch: the text claims performance 'on par with or surpassing SOTA,' but the tables consistently show below-SOTA numbers. And the one-shot protocol uses a 100-sample auxiliary set plus 20 exemplars, which is not the standard one-shot evaluation, so the comparison to published one-shot results is apples-to-oranges.\n\nOne more concern, which the reader flagged: the dVED reconstructs only the max-pooled heatmap UR, not the joint-aware U, so the tokens are learned under a joint-agnostic objective. This may discard joint identity, which matters for action semantics. The authors should at least discuss this.\n\nThese are fixable problems. The keypoint-wise masking idea is worth a serious look, and the experiments, while uncontrolled, cover four distinct tasks. I would not desk-reject this: I'd send it to review with an explicit request for the missing baseline, data-scale disclosure, masking ablation, and error bars. As is, the central 'foundation model' claim is not supported, but the paper is a reasonable starting point for revision. For a reading group, it's a good case study in what counts as transfer evidence. I wouldn't cite it in its current form.","headline":"A sensible keypoint-wise masking idea, but the foundation-model claim needs a from-scratch baseline and real pretraining statistics before it is credible.","tokens_in":19987,"tokens_out":2923,"would_cite":false,"duration_ms":27430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoFM claims that human motion can be encoded as discrete tokens from spatio-temporal pose heatmaps, and that masked-token pretraining yields one backbone that handles action classification, one-shot learning, and anomaly detection with…","keywords":["motion foundation model","discrete variational autoencoder","thermal cubes","MotionBook","masked token modeling","skeleton action recognition","pose anomaly detection","self-supervised pretraining"],"falsifier":"Train the same MoFM pipeline but replace the max-pooled reconstruction target $U_R$ with the full joint-aware tensor $U$, or add per-joint channels, and compare on NTU-120 action classes where different joints move in different directions (e.g., fine-grained gesture pairs such as 'write' versus 'point to phone'). If the joint-agnostic version is clearly worse, the max-pooling is discarding joint identity that the foundation model needs.","tokens_in":18869,"feed_emoji":"🏃","tokens_out":8075,"duration_ms":69878,"temperature":0.7,"pith_summary":"MoFM is an attempt to import the foundation-model recipe into human motion understanding: turn raw poses into discrete tokens, pretrain a transformer to reconstruct masked tokens, and then reuse the same backbone for many tasks. The paper builds MotionBook, a learned dictionary of 8192 motion tokens, by training a discrete variational encoder-decoder on spatio-temporal heatmaps called Thermal Cubes. The pretrained transformer is evaluated with only a fully-connected head on four tasks—action classification, one-shot action classification, self-supervised anomaly detection, and supervised anomaly detection—and reaches performance on par with many task-specific systems. The intended contribution is a task-agnostic motion backbone, not a new state-of-the-art result for any single benchmark. If the approach works as claimed, it offers a shared, scalable starting point for downstream motion applications without redesigning architectures per task.","feed_headline":"One motion backbone, four tasks, built from pose tokens","feed_subtitle":"A single pretrained transformer adapts to action recognition, one-shot, and anomaly detection with only a linear head.","key_machinery":"The central object is the Thermal Cube: a spatio-temporal patch of pose heatmaps, created by placing a Gaussian at each 2D joint per frame and max-pooling over all joints into the reduced map $U_R$. The dVED, a discrete variational encoder-decoder using Gumbel-Softmax, converts each cube into one of 8192 discrete tokens, forming the MotionBook vocabulary; the encoder acts as the tokenizer for the downstream transformer. The MoFM backbone is a 12-layer vision transformer with a spatio-temporal embedding layer, pretrained to predict masked token IDs under keypoint-wise masking (Algorithm 1), which targets cubes containing randomly sampled joints rather than random blocks. The machinery works by forcing the transformer to predict which motion tokens are hidden, thereby learning spatial and temporal regularities of human movement in a task-agnostic way.","core_discovery":"The paper's central claim is that a single pretrained motion transformer can serve as a foundation model for diverse human-motion tasks once poses are converted into a discrete token vocabulary. The conversion operates on spatio-temporal heatmaps built from 2D skeletons (Thermal Cubes); a Gumbel-Softmax discrete variational encoder-decoder learns the vocabulary, MoFM's dVED, and the resulting MotionBook tokens are used as targets for BERT-style masked prediction. Pretraining uses keypoint-wise masking rather than random block masking, because random blocks frequently cover regions with no skeleton and produce poor reconstructions. After pretraining, the transformer backbone plus a small task head achieves results on NTU-60/NTU-120 action recognition, one-shot action recognition, ShanghaiTech anomaly detection, and UBnormal supervised anomaly detection that are comparable to many purpose-built models. The paper presents MoFM as a flexible, task-agnostic foundation rather than as a competitor for state-of-the-art on any single benchmark.","pith_inferences":["Because the dVED reconstructs only the max-pooled map $U_R$, joint identity is not directly encoded in the tokens; tasks that hinge on which joint moves (e.g., distinguishing left-hand from right-hand gestures) may expose this limitation, and a variant that preserves per-joint channels could be tested directly against MoFM.","The ghosting trails seen in dVED reconstructions indicate that tokens carry implicit velocity and direction information, suggesting the same discrete vocabulary could support future-frame prediction or motion forecasting as additional pretraining tasks.","The recipe should transfer to 3D skeletons, multi-person scenes, and RGB-derived heatmaps with minimal changes, which would expand MoFM beyond 2D single-person pose inputs.","A natural scaling study is to vary the pretraining corpus size and measure downstream task accuracy, since the reported pretraining uses only the CMU Panoptic dataset."],"forward_implications":["The same pretrained MoFM backbone, augmented with a fully-connected head, supports supervised fine-tuning, one-shot learning via supervised contrastive training, and self-supervised anomaly scoring through a token-level jigsaw objective.","Because the representation is built from pose heatmaps rather than pixels, it avoids background and appearance biases that affect RGB video models.","MotionBook provides discrete, interpretable tokens that are visualized across the vocabulary, analogous to words in a language model.","Keypoint-aware masking improves representation learning over blind blockwise masking by focusing the model on locations where motion actually occurs."],"supporting_citations":[{"why":"Supplies the Gumbel-Softmax categorical reparameterization used to make token selection in the dVED differentiable.","marker":"[33]"},{"why":"Demonstrates the discrete VAE plus transformer paradigm for zero-shot generation that MoFM adapts to motion tokens.","marker":"[72]"},{"why":"Provides the masked-image-modeling objective that MoFM applies to thermal cubes; also the blockwise masking variant that failed on motion.","marker":"[7]"},{"why":"Supplies the vision transformer architecture used as the MoFM backbone.","marker":"[22]"},{"why":"Defines the BERT masked-token self-supervised training objective that MoFM's pretraining follows.","marker":"[36]"},{"why":"Establishes neural discrete representation learning (VQ-VAE) that motivates MotionBook's discrete latent space.","marker":"[89]"},{"why":"Provides the NTU RGB+D dataset and its cross-view and cross-subject splits used to evaluate action classification.","marker":"[78]"},{"why":"Provides the NTU RGB+D 120 dataset and its evaluation protocols for action classification and one-shot action classification.","marker":"[51]"},{"why":"Provides the ShanghaiTech Campus dataset used to evaluate self-supervised anomaly detection.","marker":"[53]"},{"why":"Provides the UBnormal dataset used to evaluate supervised anomaly detection.","marker":"[2]"}],"fun_headline_variants":["Pose tokens from thermal cubes make a universal motion backbone","One motion transformer, tokenized poses, adapts to all tasks","Discrete motion tokens enable a foundation model for human activity","MoFM: a single backbone learns from skeletons to serve many tasks","From 2D skeletons to tokens, MoFM becomes a motion foundation model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The motion tokens are learned from a max-pooled heatmap that averages away which joint produced each activation, so the vocabulary may not preserve joint identity; if a downstream task needs to distinguish, say, the left hand from the right hand, the backbone might be missing the necessary information.","fun_headline_variants_meta":{"raw":{"variants":["Pose tokens from thermal cubes make a universal motion backbone","One motion transformer, tokenized poses, adapts to all tasks","Discrete motion tokens enable a foundation model for human activity","MoFM: a single backbone learns from skeletons to serve many tasks","From 2D skeletons to tokens, MoFM becomes a motion foundation model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000865,"raw_usage":{"total_tokens":3726,"prompt_tokens":895,"completion_tokens":2831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2742}},"tokens_in":511,"tokens_out":2831,"duration_ms":20522,"temperature":1.0,"reasoning_tokens":2742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:21:40.170518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same MoFM pipeline but replace the max-pooled reconstruction target $U_R$ with the full joint-aware tensor $U$, or add per-joint channels, and compare on NTU-120 action classes where different joints move in different directions (e.g., fine-grained gesture pairs such as 'write' versus 'point to phone'). If the joint-agnostic version is clearly worse, the max-pooling is discarding joint identity that the foundation model needs.","supporting_citations":[{"cited_title":"Zero-shot text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Demonstrates the discrete VAE plus transformer paradigm for zero-shot generation that MoFM adapts to motion tokens."},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Establishes neural discrete representation learning (VQ-VAE) that motivates MotionBook's discrete latent space."},{"cited_title":"Shahroudy, J","cited_arxiv_id":null,"evidence_quote":"Provides the NTU RGB+D dataset and its cross-view and cross-subject splits used to evaluate action classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NTU RGB+D 120 dataset and its evaluation protocols for action classification and one-shot action classification."},{"cited_title":"Future frame prediction for anomaly detection–a new base- line","cited_arxiv_id":null,"evidence_quote":"Provides the ShanghaiTech Campus dataset used to evaluate self-supervised anomaly detection."}],"review_version":1}