{"id":"ea7487d3-7afa-43f5-9cac-b7dbbedb3564","arxiv_id":"2602.02401","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A single MLLM trained with a vision-guided hybrid VQ-VAE tokenizer reports state-of-the-art or competitive results for 3D pose estimation, motion prediction, and motion in-betweening on Human3.6M and 3DPW.","lead":"Superman trains one large vision-language model to read 3D human motion from video and generate 3D skeleton sequences for prediction and in-betweening, using a shared 'motion vocabulary' built from paired video and skeleton data. A generalist reader might care because it suggests perception and generation of human motion can live in one model instead of separate systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No explicit loss aligns visual and skeletal codebook prototypes; cross-modal token agreement is never measured, so the unified motion-language claim lacks its key evidence.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: the hybrid codebook's paired visual and skeletal prototypes are only weakly coupled through the shared argmin in Eq. 2. I agree this is the most serious concern because it targets the paper's core novelty rather than merely its benchmark reporting. The provided quantitative analyses—code usage, pairwise cosine similarity, semantic-sphere visualization—are all about the learned skeleton code space; none of them demonstrates that a video-only encoding and a skeleton-only encoding of the same motion yield the same token. Without that, the claim of a 'unified, cross-modal motion vocabulary' is unsupported, and the single-MLLM story reduces to a multi-task skeleton-token language model with an auxiliary visual training signal. The concern is testable and non-speculative: the authors already have all components needed to compute cross-modal token agreement. Other issues, such as the non-standard coordinate protocol and the use of different model variants in Table 2, are real empirical threats but would weaken the SOTA comparisons without invalidating the proposed framework's conceptual contribution. Because the same concern motivated the reader's CONDITIONAL verdict, I recommend no change to that verdict.","tokens_in":19963,"tokens_out":10206,"duration_ms":105527,"concrete_test":"Using the released or re-trained VGMT on held-out Human3.6M, take N paired video/skeleton windows. Encode each modality, then select tokens using (a) skeleton-only distance ||z_s - c_s_k||^2 and (b) visual-only distance ||z_v - c_v_k||^2. Report top-1 token agreement and the MPJPE between the skeletons decoded from the two selected tokens. Also report the same agreement for the joint Eq. 2 selection against each single-modality selection. If visual-only selection agrees with skeleton-only selection at chance level (about 1/8192) or the decoded skeletons differ by much more than the VGMT reconstruction error, perception and generation do not share the claimed vocabulary, and the central novelty fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Vision-Guided Motion Tokenizer (VGMT) creates a single cross-modal motion vocabulary shared by perception (video-to-tokens) and generation (skeleton-to-tokens). The only coupling between the visual and skeletal codebooks is the shared argmin in Eq. 2, and the only training signal on the visual side is the beta_v commitment term in Eq. 3. Reconstruction error is computed on X from c_s only; c_v is never decoded. Thus there is no loss that forces the token selected from a video clip alone to match the token selected from the corresponding 3D skeleton, nor any term explicitly aligning c_v_k and c_s_k beyond their being updated on the same windows. If the two encoders drift, the pose-estimation branch predicts tokens from visual features while the generation branch consumes skeleton tokens, and the 'unified motion language' is effectively two disjoint vocabularies. The paper reports codebook utilization and pairwise similarity (Fig. 7) and 'semantic sphere' visualizations (Fig. S13), but never measures cross-modal retrieval consistency. This is not merely a missing nicety: the unified-vocabulary claim is the paper's stated novelty, and Table 8's ablation shows visual grounding helps but does not show the same token is produced from video and skeleton for the same motion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Superman, a single MLLM-based framework that performs 3D pose estimation from video, skeleton-based motion prediction, and motion in-betweening. The key novelty is a Vision-Guided Motion Tokenizer (VGMT), a VQ-VAE with a hybrid codebook of paired visual and geometric prototypes, intended to create a unified cross-modal motion vocabulary. The MLLM is then trained to autoregressively emit skeleton tokens conditioned on video, text, or skeleton inputs. Experiments on Human3.6M and 3DPW report state-of-the-art or competitive MPJPE/N-MPJPE on all three tasks, together with ablations and visualizations.","tokens_in":20307,"tokens_out":12416,"duration_ms":120658,"significance":"If the central claims hold, the paper would be a meaningful step toward unifying motion perception and generation in one architecture: a single model performing video-to-3D-skeleton estimation, skeleton motion prediction, and in-betweening, with a visually grounded discrete motion vocabulary and strong zero-shot generalization from Human3.6M to 3DPW. The overall architecture is plausible, and the paper includes useful ablations (VSA, MAFT, fusion weights, multi-task training) and efficiency analyses. The main weakness is that the core cross-modal vocabulary claim is not directly evidenced: the training objective and the reported analyses do not show that video-only and skeleton-only inputs produce the same tokens for the same motion. There are also several internal numerical and dimensional inconsistencies that currently undermine the empirical case.","major_comments":[{"comment":"The paper's central novelty is that VGMT creates a unified cross-modal motion vocabulary, but no experiment directly tests cross-modal consistency. In Eq. (3), the reconstruction loss uses only the skeletal code c_s; the visual code c_v is never decoded. The only coupling between the visual and skeletal branches is the shared argmin in Eq. (2) and the two commitment losses. There is no explicit loss aligning c_v_k with c_s_k, nor any reported metric for whether the same motion window yields the same token from video-only and skeleton-only inputs. Codebook utilization (Fig. 7) and the 'semantic sphere' visualization (Fig. S13) do not address this. Table 8 shows that joint training helps downstream pose estimation, but it does not show that the two modalities share one vocabulary. Please add either an explicit cross-modal alignment loss or a quantitative cross-modal retrieval/token-agreeme","section":"Sec. 3.1, Eqs. (2)-(3); Fig. 7; Table 8"},{"comment":"There is a dimensionality mismatch in the description of VGMT. It states that 'Both skeletal and visual features have 1024 dimensions, half of the dimensions of the codebook' while 'The Hybrid Codebook contains K=8192 codes, with a dimension of D=2048.' Then Eq. (2) evaluates ||z_v_w - c_v_k|| and ||z_s_w - c_s_k||, which requires the feature vectors and code vectors to have the same dimensionality. As written, 1024-dimensional features cannot be compared with 2048-dimensional code vectors. A projection layer or a different codebook dimension must be specified. Because Eq. (2) is the core of the tokenizer, this is essential for reproducibility.","section":"Sec. 4, Implementation Details"},{"comment":"The reported 'Avg' values for MP and MIB are not consistent with the per-horizon columns if 'Avg' is their mean. For example, in Table 2, MotionBERT MP Avg is 29.94 while (18.67+26.82+50.33)/3 = 31.94; its MIB Avg is 42.37, which is below both listed MIB columns 44.86 and 53.16. The same pattern occurs for most rows, including Superman (MP Avg 26.13 vs 27.30; MIB Avg 30.61 vs 35.56). Table 4 shows similar discrepancies. If 'Avg' is a different aggregate (e.g., averaged over all test frames), that definition must be stated explicitly; if it is meant as the mean of the listed columns, the numbers must be corrected. The SOTA claims rely on these aggregates, so this needs to be fixed before the results can be accepted.","section":"Tables 2 and 4"}],"minor_comments":[{"comment":"Typo: '3PDW' should be '3DPW'.","section":"Sec. 1"},{"comment":"The caption says 'SiC [19]' but Skeleton-in-Context is reference [30]; Human-in-Context is [19]. Please correct the citation.","section":"Fig. 6 caption"},{"comment":"The claim that MAFT adds '<0.2% extra parameters' is inconsistent with Table 5, which reports MAFT parameters of 25.45M against a total of 9605M, i.e., about 0.27%. Please reconcile the percentage.","section":"Sec. 4.1 and Table 5"},{"comment":"Please clarify the training protocol for the published baselines used in Tables 2 and 4. The text states that all models are trained only on Human3.6M, but it is not stated whether these models were retrained for this comparison or whether published checkpoints (possibly trained on other data) were used. Since the tables are used to support zero-shot and SOTA claims, this should be explicit.","section":"Tables 2 and 4"},{"comment":"The notation 'Fused: β_s, β_v = 0.3,0.7' is ambiguous; it should be written as '(β_s=0.3, β_v=0.7)' for clarity.","section":"Sec. 4.3 and Table 8"},{"comment":"For skeleton-only tasks, the paper does not specify how the frozen VGMT tokenizes an input when only skeleton features are available. Eq. (2) uses both z_v and z_s; the inference-time selection rule for MP/MIB must be stated.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment is appropriate. The central architecture is promising, but the cross-modal vocabulary claim needs either an explicit alignment mechanism or a direct retrieval-consistency measurement; the dimension mismatch and the inconsistent aggregate metrics are also fixable but currently block acceptance. If the authors address these points, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious within-subfield contribution, and the core idea is more solid than the stress-test note suggests, but the empirical presentation has problems that a good referee round should fix.\n\nWhat is actually new: the Vision-Guided Motion Tokenizer with paired visual/geometric prototypes goes beyond the skeleton-only tokenizers in MotionGPT/MotionGPT3, and putting video-to-skeleton pose estimation together with skeleton-to-skeleton prediction and in-betweening in one decoder-only LLM is a real step. The 3DPW generalization numbers (62.05 MPJPE vs 140+ for SiC and HiC) are dramatic if they hold. The ablations also support the architecture: visual grounding helps MP/MIB (Table 6), VSA/MAFT both matter (Table 7), unified training beats specialized models (Table 9), and the codebook utilization and semantic sphere visualizations are genuine attempts to show the tokenizer works.\n\nSoft spots, in proportion. First, Table 2 contains internal arithmetic errors: the MP Avg of 26.13 is not the mean of the listed 13.70/23.30/44.90, and the MIB Avg of 30.61 is lower than both the listed mid (35.99) and last (35.13) values, which is impossible for a mean. This is sloppy and undermines confidence in the headline comparison. Second, the coordinate/root protocol is non-standard—they deliberately do not translate the root to the origin—and the paper does not spell out how each baseline was adapted to that protocol, so the SOTA claims are hard to verify. Third, the stress-test concern is fair and should be addressed: the only coupling between visual and skeletal prototypes is the joint argmin in Eq. 2 and separate commitment losses; the paper never measures whether the same motion window produces the same token from video and from skeleton. That is not fatal—the system can still work if video-trained tokens are consistent with skeleton tokens—but the 'unified cross-modal vocabulary' headline needs either a direct retrieval-consistency measurement or a more modest claim. Fourth, no code, weights, or error bars.\n\nWho this is for: anyone working on MLLMs for human motion, pose estimation, or motion generation. It deserves a serious referee. I would send it to peer review and push hard for corrected tables, a clear protocol description, cross-modal token agreement numbers, and code release. The idea is worth engaging with; just don't take the numbers at face value yet.","headline":"A genuinely new unified motion MLLM architecture with real generalization claims, but internal arithmetic errors and missing cross-modal token-agreement evidence mean the numbers need referee scrutiny before being trusted.","tokens_in":20816,"tokens_out":3780,"would_cite":true,"duration_ms":38771,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single multimodal model can estimate 3D pose from video and generate future or missing skeleton motion using one shared, vision-grounded token vocabulary.","keywords":["3D pose estimation","motion prediction","motion in-betweening","multimodal large language model","vector quantized autoencoder","hybrid codebook","skeleton motion generation","vision-language model"],"falsifier":"Run the tokenizer twice on the same motion window — once with the raw video as input and once with the paired 3D skeleton — and check whether both produce the same token index. If cross-modal token agreement is barely above chance on the test set, the hybrid codebook is not serving as a shared language, and the pose-estimation and generation branches are effectively using different vocabularies.","tokens_in":19844,"feed_emoji":"🏃","tokens_out":5717,"duration_ms":47716,"temperature":0.7,"pith_summary":"Superman tries to show that perception and generation of human motion are not separate problems: a single multimodal large language model, given video, text, and skeleton tokens, can estimate 3D poses from video, predict future motion from past skeletons, and fill in missing frames between keyframes. The load-bearing idea is a Vision-Guided Motion Tokenizer that builds a discrete motion vocabulary where every token is anchored both to visual appearance and to 3D skeleton geometry, so reading motion and writing motion share one language. On standard benchmarks the model reports state-of-the-art or competitive errors on all three tasks, and it transfers to an unseen dataset after training on one dataset alone. If the claim holds, the fragmented pipeline of separate pose estimators and motion generators can be replaced by one generative model.","feed_headline":"One model reads and writes human motion from video and skeletons","feed_subtitle":"With one shared token vocabulary it matches or beats specialists on pose estimation, prediction, and in-betweening.","key_machinery":"Vision-Guided Motion Tokenizer (VGMT): a vector-quantized autoencoder that converts a 16-frame motion window into discrete tokens. Its hybrid codebook is the central object — each of 8,192 entries is a pair (visual prototype, skeleton prototype) — and Eq. 2 chooses the token that minimizes the joint Euclidean distance of the visual and skeleton features to the pair. A Visual-Skeleton Attention (VSA) module samples visual features adaptively around projected joints, and an optional Motion-Aware Fine-Tuning (MAFT) module injects skeletal geometry into the LLM's visual stream via cross-attention, adding under 0.2% parameters. The MLLM autoregressively predicts the token sequence, with pose esti","core_discovery":"The paper's central claim is that motion can be treated as a universal discrete language, and that a single decoder-only multimodal large language model trained on that language can unite two camps that have been separate: models that only perceive (video to text) and models that only generate (skeleton to skeleton). The discovery is the Vision-Guided Motion Tokenizer: unlike prior motion vocabularies built from skeletons alone, it trains a VQ-VAE with a hybrid codebook in which each code is a paired visual prototype and geometric prototype. Quantization selects a token by minimizing the summed distance of the current visual and skeletal features to the pair, and the model then reconstructs","pith_inferences":["Editorial inference: the same frozen tokenizer could be attached to text-conditioned generation, since the visual prototype half of each code gives text a natural anchor to the motion vocabulary; the paper trains on video/skeleton inputs and does not test text prompts, so this is a natural next check.","Editorial inference: the strong zero-shot results on unseen data may owe more to the discrete cross-modal vocabulary than to the LLM itself; a direct comparison against a continuous-regression model with the same visual front end would isolate that contribution.","Editorial inference: the hybrid-codebook pairing should be probed for cross-modal consistency; if the same motion video and its paired skeleton yield different token indices frequently, the model is likely relying on task-specific shortcuts rather than one shared motion language."],"forward_implications":["If the central claim is correct, motion perception and motion generation no longer need separate model families: one parameter-efficient fine-tuned LLM covers video-to-pose, pose-to-future, and keyframe interpolation.","Joint training on all three tasks outperforms training on each task separately, so the shared token vocabulary transfers knowledge across perception and generation.","Grounding the motion vocabulary in vision improves even skeleton-only generation tasks, meaning visual features encode motion cues that purely geometric skeletons miss.","The unified model generalizes zero-shot from one dataset to an unseen one, with the tokenizer retaining high code utilization, implying the learned vocabulary is not dataset-specific.","Scaling the LLM and the codebook both reduce pose error, so the architecture has a clear path to further gains."],"fun_headline_variants":["One motion language from video or skeletons, trained jointly","Unified model reads video and writes skeleton motion","Vision-guided tokens bridge perception and generation in one model","Single model matches specialists on all human motion tasks","Superman: one model, one tokenizer, all motion tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that minimizing the summed distance of visual and skeleton features to a paired code is enough to keep the two modalities aligned on the same motion; if the visual and skeleton halves of the codebook drift apart, the claimed unified motion language is two separate vocabularies.","fun_headline_variants_meta":{"raw":{"variants":["One motion language from video or skeletons, trained jointly","Unified model reads video and writes skeleton motion","Vision-guided tokens bridge perception and generation in one model","Single model matches specialists on all human motion tasks","Superman: one model, one tokenizer, all motion tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1183,"prompt_tokens":806,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":550,"tokens_out":377,"duration_ms":4029,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:22:54.332754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the tokenizer twice on the same motion window — once with the raw video as input and once with the paired 3D skeleton — and check whether both produce the same token index. If cross-modal token agreement is barely above chance on the test set, the hybrid codebook is not serving as a shared language, and the pose-estimation and generation branches are effectively using different vocabularies.","supporting_citations":[],"review_version":1}