{"id":"8f84b728-5d68-4a12-acc2-eca66601c77d","arxiv_id":"2501.10713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A distributed MetaHuman-based architecture combines conversational AI, camera-based user detection, and generative plus motion-captured nonverbal behavior for real-world human-agent interaction studies.","lead":"This paper describes a modular software architecture that makes a realistic Epic Games MetaHuman avatar converse autonomously with museum visitors while performing gestures and facial expressions. It reports deployment experience from a three-week field study and argues the setup can support research on nonverbal behavior in real-world settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Motion-captured gestures are not time-locked to synthesized speech, so the architecture as described cannot present controlled nonverbal stimuli.","rationale":"The reader identified latency as the weakest assumption, which is reasonable and explicitly acknowledged in Section V. My stress-test points to a more specific and internal failure mode: even with zero latency, the architecture as described does not synchronize the duration or onset of pre-recorded motion-capture gestures with variable-length TTS output. This is load-bearing because the central claim is not merely that the system can run, but that it can be used to study nonverbal behavior; a controlled study requires the nonverbal stimulus to be consistent across trials. The paper gives credit for a plausible modular design and a real three-week deployment, and it does not overclaim empirical results. However, the missing alignment layer means the conditional acceptance should remain: the authors should either document that such alignment exists, add it, or restrict the claim to scenarios where the speech script is fixed. The proposed concrete test is modest and directly checks whether the concern lands in the actual implementation.","tokens_in":6199,"tokens_out":5517,"duration_ms":66031,"concrete_test":"Add timestamp logging at the Unreal Engine level for (i) mocap clip start/end and (ii) TTS audio playback start/end; run the existing museum scenario with e.g. 50 greetings and 50 farewells (variable TTS text lengths) and compute per-trial onset asynchrony and duration ratio (clip/audio). If median duration ratio deviates from 1 by more than about 15%, or onset asynchrony exceeds 200 ms, the synchronization concern is confirmed. Then implement an alignment step that scales or retimes the mocap clip to the synthesized audio and repeat the measurement; if the deviation drops, the architecture needs that step before it can be used for nonverbal-behavior studies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the architecture can be used to study nonverbal behavior in real-world interactions. That requires the nonverbal channels to be temporally coordinated. In the described pipeline, the TTS audio drives Audio2Face (Sec. III.F: \"The resulting audio file from the TTS is sent to Audio2Face... streamed directly to the MetaHuman's face via LiveLink\"), while body gestures are pre-recorded motion-capture clips stored per knowledge-base speech act (Sec. IV.A: \"movements are tagged with an identifier... stored with the corresponding answer\"). When a response is triggered, the mocap asset is \"selected based on its ID, and displayed on the MetaHuman\" (Sec. IV.B). Nowhere is there a mechanism to retime, stretch, or start the mocap clip relative to the TTS audio, whose duration depends on the synthesized response and the chosen voice. For a fixed intent the response text and TTS length can vary, so the gesture-speech alignment is uncontrolled. Without synchronization, a user study cannot attribute differences in engagement, comprehension, or rapport to the nonverbal-behavior condition; the independent variable is confounded with timing mismatch. Section V's admitted \"system delays... can negatively impact the user experience\" is a consequence of this same fragility: even if total latency were zero, the fixed-duration mocap clip and variable-duration TTS audio would still drift in alignment. The architecture therefore needs an alignment/timeline layer (or a fixed-script constraint) before the claim \"offers a robust and adaptable framework for exploring nonverbal behavior\" is supportable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a distributed client-server architecture for a socially interactive agent embodied as an Epic Games MetaHuman, intended for autonomous verbal and nonverbal interaction in real-world settings. The system combines a hybrid NLU/LLM dialog system, camera-based user detection and distance estimation, local Speech-to-Text and cloud Text-to-Speech, Audio2Face for real-time facial animation, and pre-recorded motion-capture body gestures. The authors report insights from a three-week field deployment in the Deutsches Museum Bonn and argue that the modular design enables flexible experimental conditions for studying nonverbal behavior. No quantitative evaluation data are provided.","tokens_in":6452,"tokens_out":4363,"duration_ms":48203,"significance":"If the architecture functions as described, it addresses a real gap: most gesture-generation and ECA evaluation studies are confined to laboratory settings, and the paper explicitly targets 'in the wild' interaction research. The modular decomposition is a strength, because it permits swapping local and cloud components and creates separable experimental conditions. The use of consumer-grade hardware, local models for GDPR compliance, and a real public venue are concrete and valuable design choices. The paper also includes a machine-accessible ethical approval statement. However, the significance of the contribution depends on two things that are not established in the manuscript: temporal coordination between gesture and speech, and basic evidence that the deployed system actually supports usable interaction. As written, the claims are plausible but largely unsupported by data.","major_comments":[{"comment":"The architecture does not provide any temporal alignment mechanism between the pre-recorded motion-capture gesture and the TTS-generated speech. The gesture asset is selected by knowledge-base ID and then \"displayed on the MetaHuman,\" while the audio is synthesized independently and its duration varies with the response text and voice. For a fixed intent, the response length can vary (particularly with the LLM fallback), so the gesture-speech offset is uncontrolled. The Talking-state claim in Section III.B of \"synchronized nonverbal behaviors\" is therefore not supported by the described components. Because the paper's central claim is that the system can be used to study nonverbal behavior, the lack of a retiming/timeline layer (or a fixed-script audio condition with known durations) confounds the independent variable: any observed effect could be attributable to timing mismatch rather than to the gesture condition. Section V's admission that system delays \"can negatively impact the user experience and reduce the effectiveness of nonverbal communication\" is a symptom of the same fragility, not a substitute for a synchronization mechanism.","section":"Section IV.B / III.F"},{"comment":"The central feasibility claim that the architecture \"offers a robust and adaptable framework for exploring nonverbal behavior\" is supported only by informal field observations. No data are reported for end-to-end latency, STT accuracy, user-detection accuracy or distance-estimation error, number of interactions, distribution of interaction lengths, or any measure of user engagement. A description of a three-week museum deployment is valuable, but without at least descriptive statistics from that deployment, the claim of \"showcasing its potential\" is an assertion rather than a demonstrated result. The manuscript should include quantitative characterizations of each module's performance and basic usage statistics from the field study.","section":"Section V and VI"}],"minor_comments":[{"comment":"The phrase \"such as e.g., facial expressions and gestures\" is redundant; choose either \"such as\" or \"e.g.,\".","section":"Abstract"},{"comment":"The word \"embodiement\" is misspelled; it should be \"embodiment.\"","section":"Figure 2 caption"},{"comment":"Figure 1 includes a \"RAG\" block, but Section III.C does not explain how retrieval-augmented generation is used in the dialog system; either describe this component or remove it from the architecture diagram.","section":"Figure 1 / Section III.C"},{"comment":"The user-detection system states \"30 frames per second\" and a bounding-box-relative distance estimate, but no calibration or validation of the distance estimate is described; please add a sentence on expected accuracy or cite a validation study.","section":"Section III.D"},{"comment":"When multiple users are present, the description does not specify how the system decides whose utterance to transcribe; this is relevant for the group-interaction claim and should be clarified.","section":"Section III.E"},{"comment":"Reference [18] is cited for the MediaPipe object detection task, but no in-text citation appears at the point where EfficientDet-Lite0 is introduced; please add the citation there.","section":"Section III.D / References"},{"comment":"The statement that \"only one of 23 studies was conducted in the wild\" should be attributed more explicitly to the review in [12], including the relevant scope and date of that review.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The paper is better suited as a systems/application note than as an empirical study. The reviewers should focus on the synchronization gap and the absence of performance data, both of which are fixable in revision. The self-citation [19] is used as background for group interaction design and does not appear to be a circular-support concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this if you're planning to build an in-the-wild SIA testbed. The paper actually shipped a MetaHuman agent in the Deutsches Museum Bonn for three weeks, and the architecture is described in enough detail to be replicable: camera-based user detection with MediaPipe, hybrid Dialogflow/Llama dialog, local Whisper STT, cloud TTS, Audio2Face for lips, and mocap assets for recurrent speech acts. That integration pattern is the contribution, and it is real. The related work is on point, particularly the observation from [12] that only one of 23 gesture-generation evaluation studies ran outside the lab.\n\nThe soft spots are in proportion to what the paper claims. There is no quantitative data anywhere: no latency numbers, no recognition accuracy, no detection reliability, no user counts, no interaction-quality measure. The three-week deployment is summarized as qualitative impressions. The authors are honest about this in Section V, admitting that system delays 'can negatively impact the user experience and reduce the effectiveness of nonverbal communication, thus making the study of nonverbal behavior difficult.' That admission is buried but it is there.\n\nThe stress-test note is correct on a specific technical point. The mocap clips are selected by knowledge-base answer ID (Sec. IV.B) and played on the MetaHuman, while the TTS audio drives Audio2Face (Sec. III.F). Nothing in the description retimes or synchronizes the start or duration of the mocap clip to the synthesized speech. For fixed knowledge-base answers the text is fixed, so the clip might approximately match, but there is no stated mechanism for alignment, and TTS duration can still vary. For any controlled study of nonverbal-behavior effects, that is a confound. The paper claims the architecture 'offers a robust and adaptable framework for exploring nonverbal behavior,' but as described it cannot present controlled nonverbal stimuli. Adding a timeline/alignment layer, or constraining to fixed scripts, would fix this.\n\nThe citation pattern is fine; the one self-citation [19] is background context. No circularity.\n\nWho should read this: people who want a concrete starting point for a field-deployable MetaHuman system, or who need a citation for 'in-the-wild SIA infrastructure.' Not for anyone wanting evidence about nonverbal behavior effects.\n\nRecommendation: this deserves a serious peer review, but as a major-revision item, not as-is. The authors should either add the missing synchronization layer or reframe the paper as a deployment report rather than a framework for studying nonverbal behavior. Desk rejection would be too harsh for a paper that actually ran in a museum.","headline":"A genuinely deployed MetaHuman architecture, worth knowing about, but the paper overclaims what it can currently show about nonverbal behavior.","tokens_in":6972,"tokens_out":3395,"would_cite":false,"duration_ms":34094,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A distributed architecture lets a MetaHuman converse and gesture in the wild","keywords":["Socially Interactive Agents","Nonverbal Behavior","MetaHuman","Generative AI","Motion Capture","Field Study","Human-Robot Interaction","Gesture Generation"],"falsifier":"Measure the time from the end of a visitor's utterance to the start of the MetaHuman's spoken response in a comparable deployment; if the median exceeds roughly one to two seconds in regular operation, the system is too slow for natural turn-taking even though every component works. A second check would swap cloud TTS for a local model and re-measure, isolating whether the API round-trip is the main delay.","tokens_in":6034,"feed_emoji":"🤖","tokens_out":7024,"duration_ms":67282,"temperature":0.7,"pith_summary":"Socially interactive agents need human-like nonverbal behavior as much as words, but most gesture-generation systems are evaluated in laboratories rather than in real interactions. This paper argues that the missing piece is an architecture that can run an autonomous agent outside the lab, and presents one built around a MetaHuman embodiment. The system combines a hybrid dialog engine, camera-based user detection, local speech recognition, cloud text-to-speech, generative facial animation, and motion-captured gestures, deployed for three weeks in a public museum. The intended contribution is a reusable testbed that lets researchers compare nonverbal-behavior conditions, such as generative versus motion-captured gestures, in realistic settings.","feed_headline":"A museum MetaHuman runs a full conversational pipeline in the wild","feed_subtitle":"Local language models, camera tracking, and motion-captured gestures bring nonverbal interaction research outside the lab.","key_machinery":"The carrying mechanism is a distributed pipeline coordinated by a four-state machine that moves the MetaHuman through idle, listening, thinking, and talking states. Camera-based user management detects people and group size; a locally run Whisper model transcribes speech; a hybrid dialog system routes known queries to a knowledge-base NLU and open-ended ones to a locally run Llama 3.2 model; cloud TTS generates audio; Audio2Face turns that audio into facial animation streamed to the MetaHuman over LiveLink; and motion-captured gesture assets are matched by identifier to knowledge-base answers. REST, WebSocket, and gRPC links let each module run on a separate consumer GPU, which the paper argues reduces reaction times.","core_discovery":"The paper's central claim is that a modular distributed client-server architecture can make a MetaHuman interact autonomously with visitors through both speech and nonverbal channels, using consumer-grade hardware and a mix of locally run and cloud models. The claim is architectural feasibility rather than a measured behavioral effect: the three-week field study demonstrates that the components can be integrated and tuned, not that one animation method outperforms another. The authors report that the general structure transferred to the real-world setting, while specific modules needed on-site adjustment for noise, lighting, and user demographics.","pith_inferences":["Editorial inference: if the architecture proves stable, the cleanest next experiment is a within-subject comparison in which only the gesture source changes, since the rest of the pipeline stays identical.","Editorial inference: the cloud TTS round-trip is the likeliest latency bottleneck named in the paper; replacing it with a local voice model would directly test whether latency, rather than animation quality, limits engagement.","Editorial inference: the reported mismatch between LLM output and the agent's physical presence suggests that embodiment-aware prompting is a testable design variable, not just a prompt-engineering nicety.","Editorial inference: because camera-based user management reports group size, the same architecture could be used to study whether an agent's nonverbal greeting behavior changes for solo visitors versus groups."],"forward_implications":["Because modules are swappable, researchers can run the same MetaHuman with only generative facial animation or only motion-captured gestures and compare the two in a live interaction.","The hybrid NLU and LLM dialog keeps routine answers consistent while still handling open questions, but it needs careful prompt engineering so the LLM respects the agent's embodiment.","Deployment success depended on on-site tuning for each environment, so reuse across sites should budget for adjustment time rather than assuming zero-configuration transfer.","The system tracks the number of people present and can adapt responses for groups, opening a path to studying group interaction.","If the architecture holds, field studies of nonverbal behavior can move from laboratory benchmarks to repeated in-the-wild deployments on accessible hardware."],"supporting_citations":[{"why":"Documents the state of co-speech gesture generation and argues for evaluating methods in actual interactions, motivating the real-world field study.","marker":"[2]"},{"why":"Identifies processing time as a central challenge for gesture-generating agents, which the distributed design targets.","marker":"[3]"},{"why":"Defines the GENEA benchmark series in which motion-captured human motion serves as the baseline for generative gesture models.","marker":"[7,8,9]"},{"why":"Audio2Face is the component that converts TTS audio into the MetaHuman's facial animation in the pipeline.","marker":"[10]"},{"why":"A review finding that only one of 23 gesture-generation studies was conducted in the wild, framing the gap this architecture fills.","marker":"[12]"},{"why":"Provides the prior conversational-agent framework for gesture generation that this paper extends with user detection and a hybrid NLU and LLM dialog.","marker":"[13]"},{"why":"Supplies the Dialogflow NLU service that routes known queries from the knowledge base.","marker":"[14]"},{"why":"Supplies the Llama 3.2 model run locally as the LLM fallback for open-ended utterances.","marker":"[15]"},{"why":"The OpenAI TTS cloud service generates spoken output, providing the audio that drives both voice and facial animation.","marker":"[17]"},{"why":"The locally run Whisper model provides the speech-to-text front end for the interaction loop.","marker":"[20]"}],"fun_headline_variants":["Museum MetaHuman blends speech and mocap gestures","Autonomous MetaHuman runs nonverbal pipeline in the wild","Real-world interaction study with MetaHuman in Bonn","Generative and motion-capture for MetaHuman nonverbal behavior"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The architecture's research value depends on the time between a visitor's utterance and the agent's reply staying short enough for natural conversation, because the authors observe that delays from insufficient compute or cloud API calls degrade the experience and make nonverbal behavior difficult to study.","fun_headline_variants_meta":{"raw":{"variants":["Museum MetaHuman blends speech and mocap gestures","Autonomous MetaHuman runs nonverbal pipeline in the wild","Real-world interaction study with MetaHuman in Bonn","Generative and motion-capture for MetaHuman nonverbal behavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1765,"prompt_tokens":816,"completion_tokens":949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":886}},"tokens_in":432,"tokens_out":949,"duration_ms":9165,"temperature":1.0,"reasoning_tokens":886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:00:39.394978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the time from the end of a visitor's utterance to the start of the MetaHuman's spoken response in a comparable deployment; if the median exceeds roughly one to two seconds in regular operation, the system is too slow for natural turn-taking even though every component works. A second check would swap cloud TTS for a local model and re-measure, isolating whether the API round-trip is the main delay.","supporting_citations":[{"cited_title":"A Comprehensive Review of Data-Driven Co-Speech Gesture Generation,","cited_arxiv_id":null,"evidence_quote":"Documents the state of co-speech gesture generation and argues for evaluating methods in actual interactions, motivating the real-world field study."},{"cited_title":"Gesture Generation,","cited_arxiv_id":null,"evidence_quote":"Identifies processing time as a central challenge for gesture-generating agents, which the distributed design targets."},{"cited_title":"Audio2Face: Generating Speech/Face Animation from Single Audio with Attention-Based Bidirectional LSTM Networks,","cited_arxiv_id":null,"evidence_quote":"Audio2Face is the component that converts TTS audio into the MetaHuman's facial animation in the pipeline."},{"cited_title":"The GENEA Challenge 2023: A large-scale evaluation of gesture generation models in monadic and dyadic settings,","cited_arxiv_id":null,"evidence_quote":"A review finding that only one of 23 gesture-generation studies was conducted in the wild, framing the gap this architecture fills."},{"cited_title":"A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents","cited_arxiv_id":"2102.12302","evidence_quote":"Provides the prior conversational-agent framework for gesture generation that this paper extends with user detection and a hybrid NLU and LLM dialog."},{"cited_title":"A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents,","cited_arxiv_id":null,"evidence_quote":"The OpenAI TTS cloud service generates spoken output, providing the audio that drives both voice and facial animation."}],"review_version":1}