{"id":"51a16549-6600-4771-8455-568da0c72eea","arxiv_id":"2412.00174","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SOLAMI is an end-to-end social vision-language-action model that takes a user's speech and body motion as input and generates a 3D character's spoken and gestural responses in one pass.","lead":"This paper presents SOLAMI, a system that lets users talk and gesture with 3D animated characters in VR by feeding the user's speech and motion into a single AI model that responds with both speech and body movement. The authors also created a synthetic dataset and report that their end-to-end model beats modular pipelines on accuracy, naturalness, and speed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative support for the headline claim is generated on the synthetic SynMSI test split and scored by GPT-4o, the same model used to create the training scripts; the 60-user VR study is the only real-world evidence, leaving synthetic-to-real transfer as the central unproven assumption.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test identified the same soft spot, the reliance on synthetic SynMSI data, and sharpened it with a compounding factor: the quantitative evaluator (GPT-4o) is also the generator of the ground-truth scripts. This does not make the method implausible; it means the strongest quantitative evidence is not independent of the data-generation process. The user study partially addresses real-world transfer but is limited in size and subjectivity. The appended Future Work section corroborates the limitation. I found no internal inconsistency in the architecture or training loss, and the latency advantage of an end-to-end token model is mechanistically plausible. However, because code, data, and interaction logs are not released, the condition that real-world transfer holds is untested. This keeps the verdict CONDITIONAL: accept only if the real-data human evaluation confirms the margin, or if the claims are weakened to in-distribution synthetic performance.","tokens_in":22011,"tokens_out":8341,"duration_ms":74419,"concrete_test":"Record 30 real VR interaction sessions with the Sec. 5 Quest 3 interface, using users who are not the authors, and save the user speech and body-motion streams. Run SOLAMI and the DLP (MotionGPT) baseline on these same inputs and have independent human annotators, blind to method, rate the generated responses on the four dimensions in Table 2 (motion coherence, motion interaction, speech consistency, overall experience), reporting means and 95% CIs. If SOLAMI does not significantly exceed DLP on the primary dimensions, the synthetic-data transfer assumption and the GPT-4o judge circularity would be decisive, and the headline claim would need to be restricted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SOLAMI produces more precise and natural responses than modular LLM agents (Abstract, Sec. 6.2) rests on Table 1 and the VR user study. Table 1 is evaluated on a 10% test split of SynMSI, which is generated by the same GPT-4o script-generation and motion-retrieval pipeline used to create the training data (Sec. 4.3). This makes the test in-distribution by construction. In addition, the Context Relevance and Character Consistency metrics are computed by GPT-4o (Sec. 6.2), the same model that authored the ground-truth scripts; the SynMSI Dataset row in Table 1 scores 4.888/4.893, near ceiling, which is consistent with judge-data circularity. The only genuinely out-of-distribution evidence is the VR user study (Sec. 6.3), but 60 self-reported Likert responses, without released interaction logs or objective measures of motion understanding, do not establish that the model handles the diversity of real user motions and speech. Appendix A (Future Work) explicitly concedes that collecting real-time data of actual dyadic interaction could enable our model to generate more precise and natural body language and speech, which undercuts the generalizability of the synthetic-data evaluation. If the synthetic-to-real transfer does not hold, the quantitative margin over DLP in Table 1 would not reflect real-world performance. This is a load-bearing concern rather than a refutation: the architecture is coherent and the user study is suggestive, but the current evidence does not yet support the headline claim at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SOLAMI, an end-to-end social vision-language-action (VLA) model for driving 3D autonomous characters in immersive VR interaction. Given the user's speech and body motion as input, the model generates the character's responsive speech and motion tokens through a decoder-only LLM backbone equipped with motion and speech tokenizers. The authors also introduce SynMSI, a synthetic multimodal interaction dataset built from existing motion-text datasets, GPT-4o-generated scripts, and TTS/voice cloning, and they describe a VR interface built on Oculus Quest 3. Quantitative evaluations on a held-out split of SynMSI and a 60-participant VR user study are reported, with the central claim that SOLAMI achieves more precise and natural multimodal responses with lower latency than modular LLM-agent baselines such as DLP and LLM+Speech.","tokens_in":22284,"tokens_out":3148,"duration_ms":28843,"significance":"If the reported results hold, the paper advances an underexplored direction: applying the end-to-end VLA paradigm, which has proven successful in robot manipulation, to social interaction with virtual characters. The main strengths are the coherent unified architecture, the automatic synthetic data pipeline that repurposes existing motion datasets, and the tangible VR testbed for cross-architecture comparison. The paper also gives useful ablations (e.g., full fine-tuning vs. LoRA, pretraining on/off). However, the quantitative evidence for the headline claim is weakened by the evaluation design: the test split is drawn from the same synthetic pipeline that produced the training data, the GPT-4o-based automatic judge belongs to the same model family used to generate the ground-truth scripts, and the reported averages over 5 runs lack variance estimates and statistical tests. The VR user study is the only out-of-distribution evidence, but it is limited to self-reported Likert scores from 60 participants. These issues are fixable in a revision, and the architecture and system contributions remain potentially valuable.","major_comments":[{"comment":"The claim of 'significantly outperforms' is not supported by the reported statistics. Table 1 reports only point averages over 5 runs, with no standard deviations, confidence intervals, or significance tests for either the motion metrics or the speech metrics. For example, the difference between SOLAMI (full params) and DLP in FID is 4.254 vs. 3.443, but without variance or a paired test it is impossible to know whether this is meaningful. Please report per-run results or error bars and run appropriate statistical tests (or at least justify that the differences exceed run-to-run variability).","section":"Sec. 6.2, Table 1"},{"comment":"There is a circularity concern in the speech-content evaluation. GPT-4o is used to generate the SynMSI dialogue scripts (Sec. 4.3) and is also used as the judge for Context Relevance and Character Consistency (Sec. 6.2). The near-ceiling scores for the SynMSI Dataset row (4.888 and 4.893) are consistent with this judge-data overlap. I request either a human evaluation with inter-annotator agreement, an independent judge from a different model family, or at least a calibration study showing that GPT-4o as judge agrees with human ratings on a held-out sample.","section":"Sec. 4.3 and Sec. 6.2"},{"comment":"The baseline comparisons raise fairness questions that should be addressed explicitly. DLP is modified by replacing its MoMat-MoGen module with MotionGPT because the original was 'too slow for user interaction (over 5 seconds latency)', and LLM+Speech truncates responses to at most 3 sentences. The resulting latency comparison (2.639s vs. 5.518s) is therefore between SOLAMI and a modified DLP, not the published DLP system. Please report the latency of the original components, justify the truncation policy's effect on speech-quality scores, and discuss how these changes affect the conclusions.","section":"Sec. 6.1 and Sec. 6.2"},{"comment":"The paper's own future-work section concedes that 'collecting real-time data of actual dyadic interaction could enable our model to generate more precise and natural body language and speech.' This is directly relevant to the synthetic-to-real transfer assumption. Since the quantitative evaluation in Table 1 is on a split of the synthetic SynMSI data, and the only out-of-distribution evidence is a 60-participant self-report study, the paper should temper its generalizability claims or provide additional evidence of transfer (e.g., interaction logs, objective measures of motion understanding in VR, or a larger and more diverse user study).","section":"Appendix A and Sec. 6.3"},{"comment":"The claim of being 'the first end-to-end Social vision-Language-Action' should be reconciled with prior work cited in the paper itself, especially body-of-her [9], which appears to describe an end-to-end humanoid agent. The related-work section discusses modular LLM agents and motion LLMs but does not position SOLAMI against this prior end-to-end effort. Please clarify the precise technical differences or adjust the 'first' claim.","section":"Abstract and Sec. 2"}],"minor_comments":[{"comment":"There is a typo in the second sentence: 'foundamental' should be 'fundamental'. Also, 'an immersive' appears twice in the abstract; please rephrase.","section":"Abstract"},{"comment":"The hyperparameters lambda_r, lambda_e, lambda_c, and lambda_v are described only as 'manually adjusted weights'. For reproducibility, please provide their concrete values or a link to the configuration.","section":"Sec. 3.2, Eq. (2)"},{"comment":"The speech tokenizer is described as SpeechTokenizer, but Sec. 3.2 says the pre-trained checkpoint is taken from AnyGPT. Please clarify the relationship: is the SpeechTokenizer initialized from AnyGPT's checkpoint for the semantic layer, or are they separate components?","section":"Sec. 3.1"},{"comment":"The questionnaire dimension is called 'Speech Consistency' in Table 2 but 'Speech Coherence' in the running text of Sec. 6.3. Please unify the terminology.","section":"Sec. 6.3, Table 2"},{"comment":"The dashes in Table 1 are explained implicitly, but it would be helpful to add a footnote clarifying that methods without motion output are not evaluated on motion metrics (FID, diversity, PA-MPJPE, angle error), and that the SynMSI Dataset row serves as the reference ground-truth quality.","section":"Table 1"},{"comment":"The VR interface uses Oculus Quest full-body tracking [73], but the description could benefit from a few sentences on how the captured pose data is retargeted to SMPL-X and how the character's motion is retargeted back to the 3D avatar; this is important for reproducibility of the user study.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industrial lab and the evaluation is somewhat self-contained; I would encourage the editor to weigh the reproducibility angle: no code or dataset is promised in the submission, and the SynMSI dataset is not released. The 'first end-to-end social VLA' claim should be carefully checked by the editor against concurrent work such as body-of-her [9]. The core architecture is plausible and the VR user study is a strength, but the quantitative evaluation needs substantially more rigor before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on SOLAMI. The real contribution is the architecture: one decoder-only LLM that ingests user speech and motion tokens and emits character speech and motion tokens in a single autoregressive pass. That is genuinely new in the social-agent space, and the three-stage training (tokenizer, modality alignment, instruction tuning) is sensible. The SynMSI data pipeline is also a real asset - using GPT-4o to write scripts, retrieving motions from existing datasets, and TTS voice cloning to build a 6.3K multi-turn multimodal dataset is a practical answer to data scarcity. The VR interface and 60-person user study show the system works end-to-end, and the study reports 95% confidence intervals.\n\nThe soft spots are in the quantitative evaluation. Table 1 reports 5-run averages with no variances or significance tests, so \"significantly outperforms\" is not actually supported. More importantly, the Context Relevance and Character Consistency scores come from GPT-4o as judge, and GPT-4o also generated the SynMSI scripts - so the judge is evaluating on its own distribution. The SynMSI test split is in-distribution by construction, with near-ceiling scores (4.888/4.893) that are consistent with judge-data circularity. That does not make the result wrong, but it means the Table 1 margins over DLP should not be taken at face value. The user study is the only out-of-distribution evidence, and it is suggestive rather than conclusive at n=60.\n\nI'd also like to see the \"first\" claim squared with Body of Her [9], which is cited but not discussed; the paper should explain what distinguishes SOLAMI from that work. And no code, data, or model weights are released, which limits reproducibility - though the paper is honest in Appendix A that real dyadic data would likely improve things.\n\nOverall, the architecture is coherent and the system is a legitimate engineering contribution. The evaluation is weaker than the claims, but the central idea is plausible. This deserves a serious referee, but the revision should require variance reporting, a human agreement check on the AI judge, and a positioned novelty discussion. I'd bring it to the reading group, and I'd probably cite the architecture and SynMSI pipeline even though I'm skeptical of the headline numbers.","headline":"A real end-to-end social VLA system with a useful synthetic data pipeline, but the headline evaluation is weakened by an in-distribution test set and an AI judge drawn from the same model family that generated the data.","tokens_in":22912,"tokens_out":2672,"would_cite":true,"duration_ms":24300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SOLAMI claims to be the first end-to-end social vision-language-action model that takes a user's speech and body motion as input and generates a 3D character's speech and body motion as output, with lower latency than modular LLM-agent…","keywords":["social vision-language-action model","3D autonomous characters","multimodal interaction","VR interface","motion tokenization","speech tokenization","synthetic dataset SynMSI","multimodal dialogue"],"falsifier":"Collect a held-out set of real user–character interactions in VR (user speech, tracked body motion, and character responses) and compare SOLAMI with the DLP baseline on motion FID, PA-MPJPE, speech relevance, and end-to-end latency; if SOLAMI's advantage on the synthetic SynMSI test set shrinks or reverses on this real-data evaluation, the central claim that the synthetic-data-trained end-to-end model transfers to live interaction would be contradicted.","tokens_in":21738,"feed_emoji":"🤖","tokens_out":7043,"duration_ms":56780,"temperature":0.7,"pith_summary":"This paper tries to establish that immersive social interaction with a 3D character can be cast as one end-to-end sequence-modeling problem rather than a pipeline of separate speech, language, and motion modules. The authors build SOLAMI, a decoder-only LLM that ingests the user's speech and motion through discrete tokenizers and autoregressively outputs the character's responsive speech and motion tokens, then decodes them to drive the avatar. Because no text intermediary is used, the model can carry body-language nuances and respond faster than modular baselines such as LLM+Speech and DLP. The authors also present SynMSI, a synthetic multimodal interaction dataset made from existing motion databases with LLM-written scripts and TTS voices, and a VR interface for live user tests. If the claim holds, lifelike social characters become a model-training problem, not a software-engineering integration problem.","feed_headline":"One model now drives 3D characters' live speech and body language","feed_subtitle":"Trained on a synthetic interaction dataset, it beats modular text-pipeline agents in naturalness and latency during VR user tests.","key_machinery":"The load-bearing mechanism is tokenization: user speech is encoded with SpeechTokenizer into semantic tokens and user motion is encoded with three VQ-VAEs for body, hands, and inter-character relative transform, turning both modalities into discrete sequences that the decoder-only LLM can treat as additional languages. The LLM then generates the character's response tokens in a multi-round conversation template marked by special modality tokens. A three-stage training scheme—tokenizer training, multi-task pre-training for motion-text and speech-text alignment, then instruction tuning on SynMSI—is what makes the token spaces usable for social dialogue, and the SynMSI pipeline supplies the interaction data by retrieving motions from text-motion databases and refining LLM-written scripts around them.","core_discovery":"The paper's central claim is that a social VLA model can replace the modular LLM-agent architecture for embodied 3D characters. SOLAMI treats user speech tokens and user motion tokens as inputs to a decoder-only LLM, predicts the character's speech and motion tokens from context and character setting, and decodes them with SoundStorm and VQ-VAE decoders. The authors report that this end-to-end design gives better motion quality (FID, PA-MPJPE, angle error), better context-relevant speech, and lower inference latency than the LLM+Speech and DLP baselines, and that a 60-participant VR user study rates it highest on motion coherence, motion interaction, speech consistency, and overall experience. The paper also claims SynMSI makes training feasible by generating 6.3K multi-turn multimodal conversations from existing motion datasets.","pith_inferences":["If the end-to-end design generalizes, the same tokenized speech-plus-motion interface could be extended to vision input (user video) once paired social interaction data exists, giving characters genuine visual perception of the user.","A direct test of the synthetic-data assumption would be training on SynMSI but evaluating with real captured VR interactions, including noisy tracking, to see whether the motion-quality and latency advantages persist outside the synthetic distribution.","The pipeline used to build SynMSI could be repurposed to generate multi-party interactions or object-mediated interactions, since it only requires a motion-text database and an LLM script generator.","The latency advantage is likely to grow as backbones scale, because the end-to-end model avoids accumulating per-module latencies."],"forward_implications":["A single decoder-only LLM can handle the full loop of understanding user speech and body motion and generating character speech and motion, so the text-based modular pipeline is not required for immersive interaction.","The end-to-end design reduces inference latency relative to ASR plus LLM plus TTS pipelines, which matters for real-time conversation.","Pre-training on motion-text and speech-text alignment tasks is necessary; SOLAMI without pretraining scores worse on both motion and speech quality.","Full-parameter fine-tuning outperforms LoRA fine-tuning for this multimodal dialogue task on the SynMSI data.","The SynMSI synthesis pipeline can produce large-scale multimodal interaction data from existing motion datasets, which addresses the data scarcity bottleneck for social VLA training."],"supporting_citations":[{"why":"GPT-4o writes the multimodal dialogue scripts and refines them around retrieved motions, which makes SynMSI possible.","marker":"[52]"},{"why":"MotionGPT supplies the VQ-VAE motion tokenizer design and motion-language pre-training recipe that SOLAMI adapts.","marker":"[38]"},{"why":"DLP is the modular LLM-agent baseline and contributes 2K motion-text pairs; its slow MoMat-MoGen module is replaced by MotionGPT for a fair comparison.","marker":"[17]"},{"why":"AnyGPT provides the pre-trained speech tokenizer, the backbone LLM, and the speech pre-training data that SOLAMI starts from.","marker":"[81]"},{"why":"SpeechTokenizer encodes speech into semantic tokens used as the LLM's input representation.","marker":"[84]"},{"why":"SoundStorm decodes the generated speech tokens into speech, with voice cloning from a short character voice prompt.","marker":"[13]"},{"why":"XTTS v2 performs TTS and voice cloning for SynMSI generation and for the LLM+Speech baseline.","marker":"[19]"},{"why":"Llama2-7B is the decoder-only backbone of SOLAMI and the LLM used in the LLM+Speech baseline.","marker":"[69]"},{"why":"Inter-X contributes 20K interactive motion-text pairs and 10K two-person motion pairs used in pre-training and in SynMSI.","marker":"[76]"},{"why":"HumanML3D contributes 24K motion-text pairs used for motion-text alignment and motion retrieval in SynMSI.","marker":"[29]"}],"fun_headline_variants":["End-to-end social VLA gives 3D avatars live speech and motion","Synthetic data trains social brain for 3D characters","VR users prefer end-to-end AI for natural 3D interaction","One social model beats modular pipelines for 3D characters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic SynMSI dataset—motions retrieved from existing databases with GPT-4o-written scripts and TTS voices—captures real user interaction well enough that training on it transfers to live VR use; if the synthetic distribution does not transfer, the reported gains over the baselines would not reflect real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end social VLA gives 3D avatars live speech and motion","Synthetic data trains social brain for 3D characters","VR users prefer end-to-end AI for natural 3D interaction","One social model beats modular pipelines for 3D characters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2973,"prompt_tokens":932,"completion_tokens":2041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1979}},"tokens_in":548,"tokens_out":2041,"duration_ms":14958,"temperature":1.0,"reasoning_tokens":1979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:38:59.263505+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of real user–character interactions in VR (user speech, tracked body motion, and character responses) and compare SOLAMI with the DLP baseline on motion FID, PA-MPJPE, speech relevance, and end-to-end latency; if SOLAMI's advantage on the synthetic SynMSI test set shrinks or reverses on this real-data evaluation, the central claim that the synthetic-data-trained end-to-end model transfers to live interaction would be contradicted.","supporting_citations":[{"cited_title":"Motiongpt: Human motion as a foreign language","cited_arxiv_id":null,"evidence_quote":"MotionGPT supplies the VQ-VAE motion tokenizer design and motion-language pre-training recipe that SOLAMI adapts."},{"cited_title":"Anygpt: Unified multimodal LLM with discrete sequence modeling","cited_arxiv_id":null,"evidence_quote":"AnyGPT provides the pre-trained speech tokenizer, the backbone LLM, and the speech pre-training data that SOLAMI starts from."},{"cited_title":"Inter-x: Towards versatile human-human interaction analysis","cited_arxiv_id":null,"evidence_quote":"Inter-X contributes 20K interactive motion-text pairs and 10K two-person motion pairs used in pre-training and in SynMSI."},{"cited_title":"Generating diverse and natural 3d human motions from text","cited_arxiv_id":null,"evidence_quote":"HumanML3D contributes 24K motion-text pairs used for motion-text alignment and motion retrieval in SynMSI."}],"review_version":1}