{"id":"edbe8b06-b489-4e01-ba82-159137e32c69","arxiv_id":"2411.16805","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A language model trained on continuous motion features, video, and text with text-guided frame selection improves scores on motion understanding benchmarks.","lead":"LLaMo is an AI assistant that feeds human motion data, such as skeleton positions, directly into a language model instead of converting it into words first. The system also accepts video and answers questions about actions, scoring higher than several earlier motion-language models on benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central attribution to native motion is untested: LLaMo trains on extra data and no same-data, same-backbone ablation varies only the motion representation.","rationale":"The strongest claim is that maintaining motion as a continuous modality, rather than tokenizing or translating it, improves motion understanding. The experiments that would establish this require a controlled comparison in which only that representational choice changes. The paper does not provide one. Section 4.1 describes a broad training mixture for LLaMo, including a custom 20,000-video Swing dataset and motion-text corpora, while Section 4.2's Mo-RepCount comparison explicitly states baselines were trained 'without using any additional data,' conceding an unequal-data comparison. On MoVid-Bench and BABEL-QA, baseline results are taken from prior papers and not retrained on LLaMo's mixture, so the observed gains could come from more data, the video feature enhancement, the Cross Talker, or the LLM backbone rather than from continuous motion. The paper still attributes the gains to the native representation in Section 4.2. This is not an external disagreement about consensus; it is an internal gap between claim and evidence. A matched ablation with tokenized motion on the same data would settle the point. The reader's conditional verdict is appropriate; my concern reinforces it rather than moving it.","tokens_in":11419,"tokens_out":4853,"duration_ms":45995,"concrete_test":"Run a matched ablation on MoVid-Bench-Motion: train the identical LLaMo pipeline on the identical data mixture (MoVid + Swing + HumanML3D + KIT-ML + Mo-RepCount) with the sole variation being the motion representation—continuous features as in LLaMo versus a MotionGPT-style VQ-VAE discrete codebook fed as text-like tokens—keeping frozen encoders, LLM, Cross Talker, and training schedule fixed, with at least three seeds. If the continuous version does not beat the tokenized version by more than seed variance (e.g., >2 points on MoVid-Bench-Motion 'All' accuracy), the paper's causal attribution to native motion is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 4.1–4.2 compare LLaMo against MotionGPT/MotionLLM and specialized counting models without controlling for training data or architecture. LLaMo was trained on MoVid, a custom 20,000-video Swing set with expert QA, HumanML3D, KIT-ML, and Mo-RepCount, while Mo-RepCount baselines were trained, by the paper's own statement, 'without using any additional data.' On MoVid-Bench and BABEL-QA the baselines are previous published numbers, not re-trained on LLaMo's mixture. The reported gains are therefore confounded with data scale and diversity, the video feature enhancer, the Cross Talker module, and the LLM backbone. Section 4.2 nonetheless states that performance is 'primarily attributed to its preservation of critical motion-specific nuances,' i.e., the native continuous representation. Without a matched ablation where only the motion representation changes, the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents LLaMo, a large language and human motion assistant that processes raw motion sequences and video without converting motion into language tokens. The framework includes a motion estimator, motion and video encoders, a feature enhancement module that distills video information into motion features, a Cross Talker module that performs text-guided frame selection and bidirectional cross-modal fusion, and a behavior generation module. Training uses MoVid, a custom Swing dataset, HumanML3D, KIT-ML, and Mo-RepCount. Evaluations on MoVid-Bench, BABEL-QA, the Swing set, and Mo-RepCount report improvements over MotionGPT and MotionLLM, leading the authors to claim that preserving native motion representation yields state-of-the-art motion understanding.","tokens_in":11549,"tokens_out":4816,"duration_ms":41814,"significance":"The idea of keeping motion as a continuous modality rather than tokenizing or translating it into language is timely and plausible; if substantiated, LLaMo would provide a useful design point for human-centric multimodal LLMs. The paper also addresses efficiency through text-guided frame selection, a practical concern. However, the current evidence does not establish the central attribution claim: comparisons are confounded by training data, architecture, and backbone differences, and there are no ablations or error bars. The manuscript is therefore a promising proposal rather than a validated system. On the positive side, the paper is clearly written and the proposed pipeline is well-motivated, with the main open questions being experimental rather than conceptual.","major_comments":[{"comment":"The claim in Section 4.2 that LLaMo's performance is 'primarily attributed to its preservation of critical motion-specific nuances' is not supported by the comparisons shown. LLaMo is trained on MoVid, the custom Swing set, HumanML3D, KIT-ML, and Mo-RepCount, while the MotionGPT and MotionLLM baselines are previously published results that were not retrained on the same mixture. A same-data, same-backbone comparison that varies only the motion representation (native versus tokenized or translated) is necessary to justify the attribution; without it, the observed gap could be due to data scale, the video feature enhancer, the Cross Talker, or the LLM backbone.","section":"Section 4.1–4.2, Tables 1–2"},{"comment":"No ablation isolates the proposed modules. The reader cannot tell whether Cross Talker's text-guided frame selection, the receptive-field regression, the adaptive aggregation, or the feature enhancer contributes to the results. For example, comparing against a variant that feeds all motion frames without selection, or one that removes the enhancer, would identify which component matters. Since these components are described as core contributions and Section 4.2 attributes performance to them, this omission is load-bearing.","section":"Sections 3.1–3.2"},{"comment":"The Swing evaluation rests on an author-created dataset and GPT-4 scoring, but the dataset construction, exact questions, and scoring protocol are deferred to a missing appendix. The manuscript reports LLaMo's accuracy relative to 'GT' (100/5) without providing any measure of agreement with professional coaches or ablating the evaluation prompt. This makes the 24.8 overall accuracy difficult to interpret and hard to reproduce.","section":"Section 4.2, Table 3"},{"comment":"The Mo-RepCount comparison is not controlled. The text states that all state-of-the-art models were trained 'without using any additional data,' while LLaMo was trained on the same Mo-RepCount plus MoVid, HumanML3D, KIT-ML, and 20,000 Swing videos. Additionally, reference [44] 'RepNet' cites a 3D human pose estimation paper rather than the repetition-counting RepNet used in the table, so it is unclear which baseline was actually evaluated. The comparison should be repeated with all methods trained on identical data, and the correct baseline should be cited.","section":"Section 4.2, Table 4"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any benchmark. Several gaps are small (e.g., BABEL-QA overall 0.458 versus 0.436 for MotionLLM, and some MoVid-Bench score differences are below 0.1). Without variance estimates, the claimed state-of-the-art results are not statistically supported.","section":"Tables 1–4"}],"minor_comments":[{"comment":"The sentence 'which makes LLaMo is a general human motion assistant' should read 'which makes LLaMo a general human motion assistant.'","section":"Section 3.1, first paragraph"},{"comment":"The column header 'Hull.' appears to be a typo for 'Hall.' (the text refers to hallucination).","section":"Table 1, bottom half"},{"comment":"The sentence 'Specifically, LLaMo achieves an overall accuracy of 24.80' is missing a period before 'The superior performance...'.","section":"Section 4.2, Professional Sports Analysis"},{"comment":"The max-pooling aggregation of attention weights should specify how ties are broken and whether the selection is deterministic under re-runs.","section":"Equation (2)"},{"comment":"Several citations are inconsistent, e.g., MotionCLIP-M and MotionCLIP-R are cited as [41] (OpenAI GPT-3.5), and reference [44] does not match the RepNet baseline; these need to be corrected.","section":"Tables 2 and 3"},{"comment":"The description of the feature enhancement module references an Appendix for detailed design, but no appendix is present; the relevant equations and hyperparameters (K, N, S_n, r_k) should be in the main text or a provided appendix.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid: the paper's central attribution is currently untestable because of confounds. I also noted citation errors (RepNet reference) and the absence of an appendix despite repeated references. The paper would require substantial additional experiments – matched baselines, ablations, error bars, and a full description of the Swing dataset – before the claims are acceptable. The idea is interesting and the writing is clear, so I see this as a major-revision rather than a reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LLaMo is a competent engineering contribution that keeps motion as a continuous modality for instruction tuning, with text-guided frame selection and bidirectional cross-modal fusion. It beats MotionLLM and MotionGPT on MoVid-Bench and BABEL-QA, and the design choice is genuinely different from those baselines. But the paper's central attribution—that the native-motion representation is what drives the gains—is not established by the experiments. The baselines on Mo-RepCount were trained without LLaMo's extra data, and there are no same-data, same-backbone ablations that vary only the motion representation. The paper is worth engaging with, but it needs fairer baselines and module ablations.\n\nWhat's new: keeping motion features continuous (no quantization or language translation) and using a Cross Talker to select K viewpoint frames conditioned on the text, then aggregating local and global contexts. That combination is not in MotionLLM or MotionGPT. The Motion Enhancer that enriches motion features from video via cross-attention is also reasonable. The new Swing dataset with expert QA is a useful resource if released.\n\nWhat it does well: the experimental results on MoVid-Bench and BABEL-QA are consistent across nearly all sub-metrics, not cherry-picked. The paper also honestly reports that on Mo-RepCount LLaMo does not outperform the specialized counting models on most metrics—that's a sign of measured claims, even if the abstract overstates.\n\nSoft spots: the biggest one is the confound. LLaMo trains on MoVid, Swing (20k videos), HumanML3D, KIT-ML, and Mo-RepCount, while the Mo-RepCount baselines are trained on Mo-RepCount alone. Same for MoVid-Bench: baselines are previous published numbers, not retrained on the same mixture. So you can't attribute the gains to the native motion representation or the Cross Talker; it could be data scale, the video feature enhancer, or the LLM backbone. There are no ablations that isolate Cross Talker or the Motion Enhancer. No error bars, no statistical significance. Key implementation details are in a missing appendix. Citation errors are also present (e.g., MotionLLM is cited as [41] in Table 2 instead of [1]; RepNet reference is actually for 3D pose estimation, not repetition counting). Minor but sloppy.\n\nThe custom Swing evaluation uses GPT-4 scoring, which is a limitation but not a flaw—the authors built the dataset and are transparent about expert QA as GT.\n\nWho it's for: researchers working on multimodal instruction tuning, especially with human motion or video. A serious referee should see it because the architecture idea and the benchmark results are worth checking; the paper needs a matched ablation and released code before the central claim holds.\n\nMy recommendation: send it to peer review, but require the authors to add same-data same-backbone ablations and module ablations, and to clean up the references and appendix.","headline":"Plausible architecture, unproven attribution: LLaMo's native-motion design shows consistent gains but the experiments don't isolate the cause.","tokens_in":12111,"tokens_out":2017,"would_cite":false,"duration_ms":17732,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that keeping human motion in its native continuous form, rather than converting it to language tokens, lets large language models understand complex human behavior better, and it supports this with a new framework called…","keywords":["human motion instruction tuning","multimodal large language model","motion understanding","text-guided frame selection","motion-video fusion","continuous modality preservation","sports analytics","BABEL-QA"],"falsifier":"A controlled experiment that trains LLaMo and a tokenized equivalent (e.g., MotionGPT-style) on the exact same data with the same backbone and size, then compares them on MoVid-Bench-Motion or BABEL-QA; if the tokenized variant matches or beats LLaMo, the native-motion claim is refuted. A simpler version is ablating the Cross Talker module and seeing whether performance collapses.","tokens_in":23,"feed_emoji":"🏃","tokens_out":5159,"duration_ms":93262,"temperature":0.7,"pith_summary":"LLaMo, a framework for human motion instruction tuning, is built on the claim that encoding motion as tokens or translating it into text discards fine-grained spatial-temporal detail needed for understanding complex behavior. Instead of translation, LLaMo feeds raw motion (and, when needed, motion estimated from video) into a large language model, using a Cross Talker module to select the motion frames most relevant to the query text and fuse them with it. The paper reports state-of-the-art results on MoVid-Bench-Motion, BABEL-QA, and a professional baseball/golf swing dataset, arguing that keeping motion native makes the model better at body-part awareness, reasoning, and fine-grained detail such as repetition counts. If correct, this would establish a design principle for multimodal assistants in motion-intensive domains like sports analytics and behavioral monitoring.","feed_headline":"Native motion beats text tokens for AI action understanding","feed_subtitle":"LLaMo feeds raw motion directly into a language model, topping motion Q&A and swing-analysis benchmarks.","key_machinery":"The load-bearing component is the Cross Talker module, a text-guided attention mechanism that performs language-guided frame selection, adaptive contextual aggregation, and bidirectional motion–text fusion. Given enhanced motion features and text embeddings, it computes a cross-attention matrix, max-pools it over text tokens to score each motion frame, picks the top $K$ frames, enriches each selected frame with local window attention and segment-level global attention, and then applies cross-attention in both directions before concatenating the features. This mechanism carries the paper's central claim: it keeps motion native while making it computationally efficient and semantically aligned with the query.","core_discovery":"The central discovery is that a large language model can process human motion as a continuous, independent modality without any tokenization or textification step, and that this preserves information that discrete encodings lose. LLaMo does this with three components: a motion estimator and enhancer that can derive motion features from video and enrich them with video context, a Cross Talker module that selects $K$ 'viewpoint frames' from the motion sequence by max-pooled cross-attention with text embeddings and then aggregates local and global context around each, and a behavior generation module that produces text from the fused motion-text representation. Evaluations show LLaMo exceeding GPT-3.5, MotionGPT, and MotionLLM on most motion-understanding metrics, scoring 0.458 overall on BABEL-QA (versus 0.436 for MotionLLM), and outperforming baselines on the professional swing dataset. The paper attributes these gains specifically to native motion processing and to text-guided frame selection, which also cuts the LLM self-attention cost from $O((L_T + T)^2)$ to $O((L_T + K)^2)$.","pith_inferences":["The same native-continuous principle likely extends to other high-rate non-linguistic modalities—audio, physiological signals, inertial sensors—where quantization to text is lossy; a testable extension is applying Cross Talker-style selection to those streams.","Because the reported comparisons vary in training data and backbone, the paper does not isolate whether the gains come from native motion or from training on the custom Swing set; a controlled same-data ablation separating these factors would either confirm or undercut the stated attribution.","The Cross Talker's frame-selection step is a generic attention-distillation idea that could be reused for long video or point-cloud sequences in any LLM-based multimodal model."],"forward_implications":["Instruction-tuned language models can accept continuous non-linguistic modalities like motion without a tokenizer, preserving detail that discrete codebooks lose.","Video-only inputs can be handled by estimating motion first, so the same model spans both motion-capture data and ordinary video.","Text-guided frame selection makes long motion sequences tractable for transformer LLMs by reducing per-layer complexity from quadratic in total sequence length to quadratic in text length plus a small number of selected frames.","The approach generalizes to specialized domains such as professional sports coaching, where expert-level motion nuance matters, and to detail-oriented tasks such as repetition counting."],"supporting_citations":[{"why":"Supplies the MotionLLM baseline, the MoVid video-motion dataset, and the MoVid-Bench evaluation suite that LLaMo is measured against.","marker":"[1]"},{"why":"Provides MotionGPT, the tokenization-based baseline that LLaMo is designed to outperform and the codebook approach the paper critiques.","marker":"[2]"},{"why":"Provides the BABEL-QA benchmark and its evaluation metric used to test motion-question answering.","marker":"[19]"},{"why":"Supplies the TransRAC baseline and the RepCount source dataset from which Mo-RepCount is filtered for repetition-count evaluation.","marker":"[38]"},{"why":"Supplies the Video-LLaVA video-understanding baseline that LLaMo compares against on MoVid-Bench-Video.","marker":"[39]"},{"why":"Supplies the KIT-ML motion-language dataset used as training data for LLaMo's motion-only modality.","marker":"[3]"}],"fun_headline_variants":["No tokens needed: LLaMo feeds raw motion to LLM","LLaMo: LLM that reads motion directly beats text-token rivals","Motion-native LLM tops benchmarks without tokenization","Cross-attention frame pick cuts LLM cost, boosts motion QA","Raw motion in, better action understanding out"],"cache_read_input_tokens":14336,"weakest_assumption_plain":"The empirical comparisons assume the performance gap is caused by LLaMo's native-motion design rather than by its larger or different training set, since LLaMo trains on additional custom data while several baselines were trained without any added data.","fun_headline_variants_meta":{"raw":{"variants":["No tokens needed: LLaMo feeds raw motion to LLM","LLaMo: LLM that reads motion directly beats text-token rivals","Motion-native LLM tops benchmarks without tokenization","Cross-attention frame pick cuts LLM cost, boosts motion QA","Raw motion in, better action understanding out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1565,"prompt_tokens":928,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":544,"tokens_out":637,"duration_ms":7216,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:09:14.419205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that trains LLaMo and a tokenized equivalent (e.g., MotionGPT-style) on the exact same data with the same backbone and size, then compares them on MoVid-Bench-Motion or BABEL-QA; if the tokenized variant matches or beats LLaMo, the native-motion claim is refuted. A simpler version is ablating the Cross Talker module and seeing whether performance collapses.","supporting_citations":[{"cited_title":"Motiongpt: Human motion as a foreign language","cited_arxiv_id":null,"evidence_quote":"Provides MotionGPT, the tokenization-based baseline that LLaMo is designed to outperform and the codebook approach the paper critiques."},{"cited_title":"Motion question answering via modular motion programs","cited_arxiv_id":null,"evidence_quote":"Provides the BABEL-QA benchmark and its evaluation metric used to test motion-question answering."},{"cited_title":"Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting","cited_arxiv_id":null,"evidence_quote":"Supplies the TransRAC baseline and the RepCount source dataset from which Mo-RepCount is filtered for repetition-count evaluation."}],"review_version":1}