{"id":"19b8c7b7-07dc-44a4-97e9-341cbd4009a2","arxiv_id":"2412.12581","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned LLaMA model, given skeleton-motion tokens, classifies emotions and generates explanatory text, with accuracy comparable to state-of-the-art skeleton classifiers.","lead":"This paper trains a large language model to classify emotions from 3D body-movement skeletons and to write a plain-language description of the emotion it detects. The system, EAI-LLM, reaches accuracy comparable to specialized skeleton classifiers while adding explanations that GPT-4 and Gemini do not match.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-skeleton/random-token control leaves the central claim that recognition is driven by skeleton tokens unmeasured; description results are also conditioned on the gold emotion label.","rationale":"The paper makes a real contribution: it demonstrates that skeleton features can be projected into an LLM token space and fine-tuned with LoRA to produce both labels and text, and Appendix B's ablation (raw encoder 46.29% vs. aligned 80%+) shows that the representation alignment matters. The issue is not that the model is vacuous, but that the specific evidence offered for the central claim does not yet isolate the causal role of the skeleton input. Because LLaMA is a strong text prior, and because the recognition prompt is highly templated, a LoRA-tuned model could achieve non-trivial accuracy from label-frequency priors alone; the paper reports no such control. Similarly, the description task is conditioned on the gold emotion label in the prompt, so the generation metrics in Table 5 demonstrate explanation of a supplied label, not explanation inferred from movement. These are both correctable with controlled experiments. I agree with the reader's CONDITIONAL verdict: the architecture is plausible and the ablations are informative, but the central claim needs the missing controls before acceptance.","tokens_in":17943,"tokens_out":6639,"duration_ms":60420,"concrete_test":"Run the same EAI-LLM training/evaluation protocol (Sections 4.2-4.3) with three additional recognition conditions on the same 4:1 split: (A) prompt with no skeleton tokens, (B) prompt with skeleton tokens replaced by fixed random noise tokens of the same shape, and (C) prompt with shuffled skeleton-token order; report accuracy per dataset alongside Table 4. If any condition reaches within, say, 5 points of the reported 85.44/71.17/66.97, the skeleton input is not load-bearing for the central claim. Additionally, for Table 5, rerun the emotion-description evaluation with the emotion label removed from the prompt and replaced by the model's own predicted label from the recognition head; if Rouge/BLEU/METEOR drop sharply, the explanation results are label-conditioned rather than skeleton-derived.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that EAI-LLM's emotion outputs are driven by the 3D skeleton tokens, not by the LLM's prior or by leakage in the prompt. The paper never reports a control in which skeleton tokens are removed, shuffled, or replaced with noise while keeping the LoRA fine-tuning and prompt identical. Appendix B partially addresses this at the encoder level (Table 7: raw skeleton encoder 46.29% vs. CE+contrastive 85% on Emilya), showing the token representation matters, but it does not isolate the LLM's contribution: a LoRA-tuned LLaMA with no informative skeleton input could still exploit label distribution and prompt template. In Section 4.3 and Table 4, recognition accuracy is attributed to skeleton understanding, but the marginal gain over a text-only or random-token baseline is unmeasured. A second related gap: the emotion-description prompt in Section 3.3 supplies the gold emotion label ('The emotion of this person is [shame], please tell me some reasons for it.'), so the Rouge/BLEU/METEOR improvements in Table 5 are for explaining a known label, not for generating an explanation from the skeleton alone; end-to-end description from predicted labels is not evaluated. If the recognition gain over text prior is small, the abstract's claim that the model understands emotional body expressions is unsupported; if the label-conditioned prompt is required, the description capability is also not demonstrated from skeleton understanding alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes EAI-LLM, a framework that feeds 3D skeleton sequences into LLaMA via a GCN-based encoder and learned linear projections, with LoRA fine-tuning, to perform emotion classification and generate textual explanations. It introduces a multi-granularity skeleton tokenizer, a unified skeleton token module for joint training across heterogeneous datasets, and a skeleton-text contrastive loss with a KL-divergence formulation. Experiments on Emilya, KDAE, and EGBM report recognition accuracies of 85.44%, 71.17%, and 66.97% respectively, which are comparable to or slightly below state-of-the-art baselines. For emotion description, the reported Rouge/BLEU/METEOR scores are higher than those of GPT-4 and Gemini 1.5 Pro prompted with videos rendered from the same skeletons. The paper includes ablations on architecture, skeleton encoder, LLM choice, output format, and training order, and closes with a limitation section that acknowledges hallucination issues.","tokens_in":18202,"tokens_out":5536,"duration_ms":44924,"significance":"If the central causal claim is established, this is a useful contribution to interpretable skeleton-based emotion recognition: it demonstrates a concrete way to bridge 3D skeleton data and LLMs, enabling both classification and textual explanation. The paper's strengths include a broad experiment suite across three datasets, a detailed ablation appendix, an honest limitations section, and a direct comparison with mainstream multimodal LLMs. However, the central attribution of recognition performance to the skeleton tokens is not yet demonstrated: no text-only or random-token control is reported, and the description evaluation is conditioned on the gold emotion label rather than the model's own prediction. These gaps are load-bearing for the paper's main claims, but they are fixable with additional experiments and clarifications, so the current value is conditional on those additions.","major_comments":[{"comment":"The central claim that emotion recognition is driven by the skeleton tokens is not isolated. There is no control condition with the same LoRA-tuned LLaMA and prompt but with the skeleton tokens removed, shuffled, or replaced by random features. Appendix B shows that raw skeleton encoder features yield 46.29% accuracy versus 85.44% with CE+contrastive alignment, but this compares different feature qualities, not the contribution of the skeleton input relative to the LLM's prior. A text-only baseline (e.g., the same prompt with no <SkeletonFeature> or with fixed random vectors) is needed to measure the marginal gain from skeleton tokens; without it, the reported accuracies in Table 4 could partly reflect the model's prior over emotion labels and prompt template. This is load-bearing for the abstract's claim that the model understands emotional body expressions.","section":"§4.3, Table 4"},{"comment":"The emotion-description prompt supplies the gold label: \"The emotion of this person is [shame], please tell me some reasons for it.\" Consequently, Table 5 evaluates explanation of a known label, not generation from the skeleton alone. An end-to-end evaluation that uses the label predicted by the recognition stage is missing. Moreover, the comparison with GPT-4 and Gemini in Table 5 needs to state explicitly whether those models also received the gold label in the prompt; the example in Figure 4 suggests they did, but the table caption and text are ambiguous. This is load-bearing for the claim that EAI-LLM generates descriptions based on its own classification results.","section":"§3.3, Table 5"},{"comment":"The table is difficult to interpret and the results undermine the \"simultaneously\" claim. The R→D rows show a catastrophic drop in recognition accuracy (the text reports relative drops of 37.06% and 61.86%), meaning that after sequential fine-tuning the model cannot perform both tasks without degradation. The paper should either present a multi-task training scheme that avoids this effect, or clearly state that the two capabilities require separately fine-tuned models. Additionally, the table layout appears misaligned (e.g., the row \"Semantic R→D\" contains entries that do not match the column structure), making it impossible to verify the reported averages and drops.","section":"§4.4, Table 3"},{"comment":"Recognition results are based on a single random 4:1 split with no error bars, repeated runs, or significance tests. For a claim of \"comparable to existing methods,\" the lack of variance reporting is a serious gap, especially because the improvements over baselines on KDAE and EGBM are small (e.g., 71.17% vs. 70.46% for CTR-GCN on KDAE). Please report mean ± std over at least three random seeds, or a paired significance test, to support the comparative claims.","section":"§4.2, Evaluation Protocols"},{"comment":"The emotion-description subset consists of 174 Emilya samples and 105 KDAE samples, with no EGBM descriptions mentioned. Yet Table 3 reports description metrics \"averaged across the three datasets.\" Please clarify how EGBM contributes to the description evaluation, or restrict the reported averages to the two datasets that actually have description annotations.","section":"§4.1, Table 3"}],"minor_comments":[{"comment":"Equation (1) is described as applying \"a mask to all skeleton tokens,\" but the actual operation is padding masking to retain original tokens and nullify padded elements. Please rephrase to clarify that the mask applies only to padded positions.","section":"§3.2, Eq. (1)"},{"comment":"The text states that all baselines were re-implemented and preprocessing was standardized, but no details are given about the re-implementation (hyperparameters, training epochs, validation splits). Please provide these details in the supplementary material to support the claim of fair comparison.","section":"§4.5"},{"comment":"The confusion matrices contain unrendered Uni* encoding sequences (e.g., \"/uni00000024/...\"), indicating a PDF encoding problem. Please replace these with proper class labels.","section":"Figure 3"},{"comment":"The bullet claiming that \"semantic tokens offer a balanced trade-off between recognition accuracy and description capability\" is hard to reconcile with Table 3, where spatio-temporal tokens show much higher Rouge and BLEU scores in the D→R condition. Please restate or qualify this conclusion.","section":"§4.4"},{"comment":"The contribution list calls EAI-LLM \"the first approach\" to use LLMs for skeleton-based emotion recognition with descriptions. This is a strong novelty claim that is not supported by a systematic comparison with prior work; consider softening it to \"to our knowledge\" and citing the most related efforts.","section":"§1, Contributions"}],"recommendation":"major_revision","confidential_remarks":"The missing no-skeleton/text-only control is the most critical issue; the paper's central causal claim is currently underdetermined. The label-conditioned description prompt also weakens the end-to-end claim, though the comparison with GPT-4/Gemini may still be informative if the same prompt is given to all models. The paper is honest about hallucinations and catastrophic forgetting, which is a positive sign. I recommend major revision rather than rejection because the missing controls and clarifications are achievable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi X,\n\nQuick take on 2412.12581. It is a competent engineering contribution: apparently the first to tokenize 3D skeletons for LLM input, and it shows respectable recognition accuracy on two of three datasets plus plausible generated explanations. The ablations are genuinely useful, especially Table 7, which shows contrastive alignment matters (raw encoder 46% vs. 80%+ after alignment). The UST masking for combining heterogeneous datasets is a simple, effective trick.\n\nThe soft spots are real but not fatal. First, there is no control with shuffled or random skeleton tokens, so the marginal contribution of the LLM prior over the skeleton signal is not directly measured. Table 7 partially addresses this, but a random-token baseline would be cleaner. Second, the description task supplies the gold emotion label in the prompt ('The emotion of this person is [shame], please tell me some reasons'). So the Rouge/BLEU improvements over GPT-4 and Gemini show the model can produce a plausible explanation given the label and the skeleton, not that it independently infers the emotion and then explains it end-to-end. That is a meaningful distinction, and the abstract oversells it slightly. Third, no error bars or repeated runs, which is limiting but typical for this kind of work. Fourth, the description annotation is small (174 Emilya, 105 KDAE samples), though enough to demonstrate the idea. Finally, comparing against zero-shot GPT-4/Gemini is not a level playing field, even if they were given the same gold label in the prompt.\n\nOn balance, the central claim—that LLaMA can serve as an emotion recognizer and explainer given skeleton tokens—is defensible. The recognition numbers are close to existing GCN methods, and the token alignment is well-motivated. The missing controls and label-conditioned description task mean the interpretation should be softened, but I do not see a load-bearing flaw. The paper is honest about hallucination and catastrophic forgetting, which adds credibility.\n\nI would send this to peer review. The authors can add the controls and reframe the description claims. It is worth a reading group meeting to discuss the modality-gap handling.","headline":"Solid engineering first pass at LLM-based skeleton emotion recognition and explanation; the recognition claim holds, but the description results are weaker than advertised because the gold emotion label is part of the prompt.","tokens_in":18770,"tokens_out":5145,"would_cite":true,"duration_ms":45409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a large language model can recognize emotions from 3D full-body skeleton data and generate textual explanations by converting the skeleton into specialized tokens.","keywords":["emotion recognition","3D skeleton","large language models","body movement","multi-granularity tokenization","contrastive learning","human-computer interaction","explainable emotion recognition"],"falsifier":"On the held-out test split, run the same prompts after replacing the skeleton-token block with a constant vector of matched length, or after randomly permuting joint coordinates within each frame; if recognition accuracy stays near the reported 85.44%, 71.17%, and 66.97% values while generated descriptions remain fluent, the skeleton input is not what carries the claimed understanding.","tokens_in":17721,"feed_emoji":"🦴","tokens_out":6301,"duration_ms":50684,"temperature":0.7,"pith_summary":"EAI-LLM treats a 3D skeleton sequence as a kind of language: a multi-granularity tokenizer turns body movement into semantic, spatial, and temporal tokens, and a masked unified token module lets datasets with different joint counts and frame lengths be trained together. The same model then both outputs an emotion label and writes a short explanation of why, using the language model's pretrained knowledge to compensate for very little labeled skeleton-description data. If the central claim holds, emotion recognition from body movement becomes explainable rather than a black-box label, which matters for human-computer interaction at a distance or without microphones.","feed_headline":"3D skeleton tokens let an LLM name emotions and say why","feed_subtitle":"One model labels body-movement emotion near dedicated GCNs and beats GPT-4 and Gemini at describing the motion.","key_machinery":"The central machinery is the Multi-Granularity Skeleton Tokenizer (MGST), which extracts a semantic token summarizing the whole body motion plus spatial and temporal tokens that preserve joint and frame detail, giving the LLM diverse evidence for generation. A Unified Skeleton Token (UST) module applies an attention mask to pad all tokens to one length, so datasets of different skeleton topologies are pooled like sentences of different lengths; a skeleton-text contrastive loss, using a CLIP text encoder and KL divergence, aligns skeleton features with language space, and LoRA fine-tuning adapts the frozen LLaMA-7B base while keeping its pretrained weights intact.","core_discovery":"The paper's central claim is that 3D skeleton sequences can be tokenized into LLM-compatible inputs so that one fine-tuned language model performs emotion classification and generates fine-grained emotion descriptions at the same time. Pre-training aligns skeleton features with text features through contrastive learning, and masking normalizes heterogeneous tokens so that the Emilya, KDAE, and EGBM datasets can be jointly trained; on the authors' re-implemented comparisons the model reaches 85.44% accuracy on Emilya, 71.17% on KDAE, and 66.97% on EGBM, close to dedicated graph-convolutional recognizers, while its descriptions score higher on Rouge, BLEU, and METEOR than GPT-4 and Gemini prompted with the same skeletons rendered as video.","pith_inferences":["The reported accuracy could partly come from the language model's pretrained prior over emotion labels rather than from the skeleton input; a control with no skeleton tokens or with shuffled joint coordinates would measure how much the motion actually contributes.","A natural extension is an automatic consistency check that verifies each generated body cue (head down, hands covering face) against the actual joint geometry, which could also mitigate the hallucination examples the paper itself reports.","The same tokenize-align-and-finetune recipe could transfer to other continuous non-text signals such as gait, gesture, or physiological streams, because the paper's alignment step is modality-agnostic in principle.","Because descriptions were trained on a small hand-labeled subset, scaling the annotation with a richer emotion-action vocabulary is a testable path to more detailed explanations."],"forward_implications":["If correct, the same model can label an emotion and explain its bodily cues from pure 3D joint coordinates, no face or voice required, which suits cameras placed far from the user.","Joint training across heterogeneous skeleton datasets becomes feasible via length masking, and the authors report an average accuracy gain of about 4.5% for semantic tokens over separate training.","The language model's prior knowledge supports fluent emotion descriptions from only about 279 manually annotated skeleton-description pairs, suggesting data-efficient explainability.","Spatio-temporal tokens are better for description quality while semantic tokens are better for recognition, so the token granularity can be chosen by task.","Training order matters: fine-tuning recognition first and description second causes a large drop in recognition accuracy, which the authors attribute to conflicting output-format demands."],"supporting_citations":[{"why":"CTR-GCN is the pre-trained skeleton encoder backbone that extracts the initial skeleton features F^s.","marker":"(Chen et al. 2021)"},{"why":"LLaMA-7B is the base large language model whose pretrained knowledge is adapted with LoRA.","marker":"(Touvron et al. 2023)"},{"why":"LoRA provides the low-rank adaptation mechanism that fine-tunes the LLM while freezing its pretrained weights.","marker":"(Hu et al. 2021)"},{"why":"The CLIP text encoder supplies the text feature space used in skeleton-text contrastive alignment.","marker":"(Radford et al. 2021)"},{"why":"ActionCLIP supplies the 'This is a [happy] person' text-template format for paired skeleton-language data.","marker":"(Wang et al. 2023)"},{"why":"The visual-instruction-tuning linear projection design justifies the linear layer mapping skeleton tokens into the language-model token space.","marker":"(Liu et al. 2023)"},{"why":"Emilya provides the 8,206-sample daily-action emotion dataset that is one of the three joint-training corpora.","marker":"(Fourati and Pelachaud 2016)"},{"why":"KDAE provides the 1,402-sample actor emotion dataset used for training and evaluation.","marker":"(Zhang et al. 2020)"},{"why":"EGBM provides the 560-sample gesture and body-movement emotion corpus used for joint training.","marker":"(Sapiński et al. 2019)"},{"why":"The survey's emotional action descriptions guided the manual annotation of emotion description data.","marker":"(Noroozi et al. 2018)"}],"fun_headline_variants":["LLM reads 3D skeletons to name emotions and describe them","Skeleton tokens teach LLM emotion labels and explanations","One LLM classifies and explains body-language emotions","3D body movement as LLM tokens for emotion recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that the model's emotion judgments are actually driven by the skeleton tokens rather than by the language model's pretrained expectations about emotion labels, because no control with text-only prompts or scrambled skeleton features is reported.","fun_headline_variants_meta":{"raw":{"variants":["LLM reads 3D skeletons to name emotions and describe them","Skeleton tokens teach LLM emotion labels and explanations","One LLM classifies and explains body-language emotions","3D body movement as LLM tokens for emotion recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1320,"prompt_tokens":924,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":540,"tokens_out":396,"duration_ms":3773,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:56:44.473978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the held-out test split, run the same prompts after replacing the skeleton-token block with a constant vector of matched length, or after randomly permuting joint coordinates within each frame; if recognition accuracy stays near the reported 85.44%, 71.17%, and 66.97% values while generated descriptions remain fluent, the skeleton input is not what carries the claimed understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CTR-GCN is the pre-trained skeleton encoder backbone that extracts the initial skeleton features F^s."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ActionCLIP supplies the 'This is a [happy] person' text-template format for paired skeleton-language data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The visual-instruction-tuning linear projection design justifies the linear layer mapping skeleton tokens into the language-model token space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Emilya provides the 8,206-sample daily-action emotion dataset that is one of the three joint-training corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KDAE provides the 1,402-sample actor emotion dataset used for training and evaluation."},{"cited_title":"A.; Kami \\'n ska, D.; Sapi \\'n ski, T.; Escalera, S.; and Anbarjafari, G","cited_arxiv_id":null,"evidence_quote":"The survey's emotional action descriptions guided the manual annotation of emotion description data."}],"review_version":1}