{"id":"9cb9b4f6-9b49-4237-a9d3-5c2b18d8d0f5","arxiv_id":"2503.15504","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GRETA is updated to perceive human gaze, touch, and speech, adapt its virtual agent's behavior through feedback loops, and animate gestures incrementally in real time.","lead":"This paper describes the latest version of GRETA, a software platform for building virtual characters that talk, gesture, and react to a person. The authors explain how the platform now senses gaze, touch, and speech, adapts its agent behavior in response, and animates gestures incrementally instead of waiting for the full sentence.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real-time loop claim in §7.2 is internally tight: the stated component times sum to exactly the 40 ms frame budget, leaving zero slack for jitter or message-passing latency, and no end-to-end measurement supports the assertion of 25 fps 'without any delay'.","rationale":"The reader's weakest assumption, the real-time integration premise, is the correct place to focus. My review sharpens it: the component times in Section 7.2 are not merely unmeasured; they sum exactly to the claimed 40 ms frame period, so the architecture as described leaves no tolerance for the very jitter and message-passing delays that any real implementation of ActiveMQ, ZeroMQ, and OSC would introduce. This makes the concern a correctness risk for the central claim rather than a preference for more benchmarks. I agree with the reader's conditional verdict because the architecture is coherent and the concern is empirical, not a demonstrated internal contradiction. The appropriate test is a concrete end-to-end latency measurement, which the paper itself does not provide. If such a test were published and passed, the central claim would be substantially supported. Until then, the verification requirement stated by the reader remains unsatisfied, so the verdict should stay conditional.","tokens_in":1163,"tokens_out":831,"duration_ms":52490,"concrete_test":"Run the IA V A pipeline in the configuration of Section 7.2 with loopback video and audio input, and log four timestamps per frame: t0 when the camera/audio frame arrives, t1 when OpenFace/openSMILE output is published, t2 when the ASAP prediction is emitted, and t3 when Ogre3D presents the corresponding agent frame. Run for at least 10,000 frames and report the median, p95, and p99 of the end-to-end latency (t3-t0), plus the fraction of frames whose total loop time exceeds the 40 ms budget. If p95 exceeds 40 ms, or if any sustained backlog forms in the message queues, the '25 fps assured without any delay' claim fails and the adaptation claim must be restated as best-effort rather than guaranteed real-time behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7.2 is the load-bearing point for the paper's central claim of an adaptive, real-time socially interactive agent. The paper states that a single system loop takes 0.04 s, composed of perception of approximately 0.03 s (OpenFace at 30 fps, openSMILE at 100 Hz), adaptive behavior generation of approximately 0.008 s via ASAP, and communication/visualization of approximately 0.002 s. If these components execute serially in one loop, the sum is exactly 0.040 s, leaving no slack for scheduling jitter, synchronization waits, garbage collection, or the latency of the ActiveMQ, ZeroMQ, and OSC message buses that connect separate processes. The sentence claiming the 25 fps frequency 'is assured ... without any delay' is an assertion with no measured latency distribution, no missed-frame count, and no end-to-end benchmark. Throughput equal to the frame period does not guarantee bounded end-to-end latency; with external perception tools and multiple message-passing hops, the difference between mean throughput and per-frame latency is exactly where responsiveness is lost. If this premise fails, the reciprocal adaptation loop described in Section 3.1 and the IA V A adaptive behavior generation in Section 7.2 degrade into delayed, non-responsive reactions rather than real-time adaptation. The manuscript also contains incomplete editorial markers, notably the untranslated note in Section 7.3.2 and missing citations such as OpenPose in Section 3.2, which reinforces that the system description is not yet a validated performance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the architecture of the GRETA platform, a SAIBA-based system for controlling socially interactive agents. It reports recent extensions aimed at making GRETA adaptive: perception of external human signals (facial action units, gaze, speech, hand tracking, proximity), interpretation modules (BERTIS, dialog acts, GPT-3), an adaptive behavior generation loop (IAVA/ASAP), an incremental behavior realizer that schedules keyframe chunks, a frame-level realizer, and a social-touch framework using VR, Leap Motion, and haptics. The paper presents the overall architecture, module descriptions, and several use cases (motivational interviewing, CBT, social skills training, social touch). The central claim is that the enhanced GRETA platform supports real-time reciprocal adaptation between the agent and the human user, with a claimed 25 fps adaptive behavior generation loop in Section 7.2.","tokens_in":15919,"tokens_out":4132,"duration_ms":38649,"significance":"If the claimed integration is real, the paper would be a useful systems contribution: it shows how perception, interpretation, adaptive generation, and incremental animation can be combined within a standard FML/BML architecture. The modular design is a strength, and the description of the incremental realizer with interrupt/resume/stop/clear operations is concrete and plausible. The paper also benefits from reusing and citing established open-source components (OpenFace, openSMILE, MediaPipe) and from naming the specific prior modules (Meaning Miner, IAVA, ASAP, AMII) on which the platform builds. However, the validation is almost entirely self-referential: there are no end-to-end measurements of the integrated system, and the only quantitative performance numbers (Section 7.3.2) are a random-forest accuracy pair with no dataset description. The real-time adaptation claim in Section 7.2 is the load-bearing assertion for the paper's central contribution, and it is currently unsupported by latency distributions, missed-frame counts, or any benchmark. The significance is therefore conditional on the authors supplying such evidence.","major_comments":[{"comment":"The claim that 25 fps \"is assured ... without any delay\" is not supported by any measurement. The listed component times (approximately 0.03 s perception, 0.008 s ASAP inference, and 0.002 s communication/visualization) sum to exactly the 0.04 s frame budget, leaving zero slack for scheduling jitter, message-bus latency (ActiveMQ, ZeroMQ, OSC), or synchronization waits; throughput equal to the frame period does not imply bounded end-to-end latency. Because the reciprocal adaptation loop of Section 3.1 rests on this real-time premise, the paper must report at least an end-to-end latency distribution, a missed-frame count, and the message-passing overhead before the central claim can be accepted.","section":"Section 7.2 (Real-time Adaptive Behavior Generation)"},{"comment":"The social-touch random forest is reported as achieving 0.91 training accuracy and 0.80 test accuracy, but the paper gives no dataset size, class distribution, collection protocol, or validation procedure; with four touch classes (hit, tap, caress, stroke), 0.80 could reflect a heavily imbalanced set or a favorable split. Please add the dataset details, per-class precision/recall, and a clear account of how the train/test split was formed.","section":"Section 7.3.2 (Social Touch, Gesture Recognition and Perception)"},{"comment":"The incremental realizer splits keyframes into starting-time chunks and schedules them in a thread, but the paper does not demonstrate that this preserves the multimodal synchrony of the original BML plan. The operations Interrupt, Resume, Stop, and Clear Thread Queue are described only at the level of API behavior, with no account of how timing constraints across modalities (e.g., gesture relative to speech onset) are maintained or violated by these operations. Provide at least a small trace or controlled experiment showing scheduled versus realized onset times for a multimodal behavior, so that the claim of \"incremental realization of nonverbal behaviours\" can be checked.","section":"Section 6.1 and Section 6.2 (Incremental Behavior Realizer)"},{"comment":"The paper claims that the integrated GRETA platform can perceive, interpret, and adapt in real time, but the only quantitative evidence is drawn from previously published components (IAVA [60], ASAP [61], AMII [58]) and from the random-forest numbers in Section 7.3.2; no integrated experiment with real sensor input, recorded sessions, or latency traces is reported. Since the contribution is the integration of these components, an end-to-end demonstration (e.g., a logged interaction with timing annotations) is needed to separate novel integration claims from properties inherited from the individual modules.","section":"Sections 1, 3.1, and 7.2 (Integrated validation)"}],"minor_comments":[{"comment":"The manuscript contains the untranslated editorial note \"¨: Deplacer à section User’s Input\"; this appears to be an instruction to the authors and should be removed or acted on.","section":"Section 7.3.2"},{"comment":"Several citations are missing placeholders: OpenPose is cited as \"[]\", and Section 3.3 has empty brackets for image schemas, dialog acts, and automatic thought; these references must be completed.","section":"Section 3.2 and Section 3.3"},{"comment":"The units in the performance paragraph are garbled in the manuscript text (e.g., \"25  \", \"0.03\", \"0.04\"); these should read frames-per-second and seconds, and the encoding artifacts should be fixed.","section":"Section 7.2"},{"comment":"The text refers to \"fig. 6\" without a clear cross-reference; ensure that the architecture figure is cited consistently where each use case is described.","section":"Section 7.4"},{"comment":"The term \"gestuary\" is used without definition; please define it explicitly (e.g., a gesture library) or use a more standard term.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"This is an architecture/system paper rather than an empirical evaluation paper, so the lack of an end-to-end user study is not itself disqualifying. However, the real-time claim in Section 7.2 is a factual assertion that must be backed by measurements; without it, the paper's central contribution is not verifiable. The heavy reliance on the same group's prior modules (refs 24, 58-61) is expected for a platform paper, but the authors should explicitly state which components are new in this submission and which are reused, and they should provide a reproducible integration test. I would not reject on novelty grounds; the incremental realizer and the frame-level realizer are potentially useful contributions if their timing behavior is substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, it is a system paper, not an empirical one: the new material is the incremental realizer and the frame-level realizer, both implementation variants of standard incremental-processing ideas, and they are described without any measured latency or throughput data. Second, the paper's flagship claim in §7.2, that the system runs at 25 fps 'without any delay', is an assertion with no benchmark behind it.\n\nWhat the paper does well: the architecture is coherent and the modular integration is real. The authors have connected a lot of pieces—OpenFace, openSMILE, Google ASR, ActiveMQ/ZeroMQ/OSC, Flipper 2.0, ASAP, and their own gesture-generation models—into one platform, and the description of how these components fit into the SAIBA framework is clear. The incremental realizer's chunk-based scheduling and the frame-level realizer's OSC interface are sensible designs. The paper is also honest about some limitations, e.g., not computing touch duration or pressure.\n\nThe soft spots are the load-bearing ones. The stress-test note is right: the 40 ms loop in §7.2 is not measured. The component times sum exactly to 0.04 s with zero slack for scheduling jitter, GC pauses, or message-bus latency, and 'without any delay' is not a result. For a system whose whole point is responsive adaptation, this is the claim that needs numbers: an end-to-end latency distribution, a missed-frame count, and ideally a comparison against a non-incremental baseline. The random-forest accuracy (0.91 train, 0.80 test) is reported without dataset details, number of participants, or cross-validation. And the validation of the platform leans heavily on the group's own prior modules (BERTIS, Meaning Miner, ASAP, AMII), which is self-referential support, not independent evidence.\n\nThere are also editorial tells: an untranslated French note in §7.3.2, missing citations for OpenPose and other items in §3.3. These are minor but signal the draft isn't yet ready as-is.\n\nWho this is for: researchers building or extending SIA platforms, especially those who want to know what GRETA can and cannot do. A serious referee should engage with it, but only if the authors are pushed to supply the missing measurements and to clean up the manuscript. I'd recommend accepting it for peer review with the expectation of major revision, not desk rejection.","headline":"A coherent architecture description of the GRETA platform, but the central real-time claim rests on an unmeasured 40 ms budget and the validation is largely self-referential.","tokens_in":16487,"tokens_out":2555,"would_cite":false,"duration_ms":21640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRETA's extended architecture closes the loop from user perception to agent behavior in real time.","keywords":["socially interactive agents","embodied conversational agents","incremental behavior realization","frame-level animation","reciprocal adaptation","multimodal perception","virtual reality social touch"],"falsifier":"Instrument a running session and timestamp each hop; if the measured time from a user's facial or audio change to the agent's displayed frame systematically exceeds one 40 ms frame period, the platform does not deliver the real-time adaptation it claims.","tokens_in":15389,"feed_emoji":"🤖","tokens_out":8461,"duration_ms":72815,"temperature":0.7,"pith_summary":"The paper sets out to establish that GRETA, a modular platform for animating virtual agents, has grown from a one-way animation engine into a closed-loop socially interactive agent system. Its extended architecture receives a human user's facial expressions, gaze, head movements, speech text and prosody, hand landmarks, and proximity; interprets those signals; plans the agent's next communicative intentions; and realizes the chosen nonverbal behaviors incrementally. Because behavior can be produced and revised at frame level rather than at whole-utterance level, the agent can respond to the user's conduct during the interaction instead of only after a completed turn. If the architecture works as described, it offers other SIA builders a single, modular, standard-pipeline substrate for studying and deploying adaptive virtual interlocutors in tutoring, therapy, and training applications.","feed_headline":"Virtual agents can now read faces and adapt mid-conversation","feed_subtitle":"One modular loop now takes perception to frame-level animation, enabling adaptive tutoring and therapy agents.","key_machinery":"The load-bearing piece is the behavior-realization layer: an Incremental Realizer that cuts keyframes into chunks grouped by starting time and schedules them on a dedicated thread, an Incremental Realizer Interaction module that can interrupt, resume, stop, or clear that stream, and a Frame-level Behavior Realizer that accepts per-time-step signals and renders upper-face action units, smiles, blinks, head movements, gaze, and mouth movements. Together they let the platform turn a prediction of the next frame's user and agent state into visible behavior at frame granularity, which is what makes the adaptation loop closed.","core_discovery":"On its own terms, the paper's central claim is that GRETA now supports the full perception, interpretation, decision, and realization loop needed for adaptive social interaction. The platform takes multimodal input from facial feature extraction, speech recognition, audio prosody extraction, hand tracking, and VR tracking; converts those signals into dialog acts, image schemas, and emotional and rapport states; lets an intention planner choose the agent's next act; and then renders it either through the classic intention-to-behavior markup pipeline, an incremental keyframe-chunk realizer that can be interrupted and resumed, or a frame-level realizer that displays predicted next-frame behavior at 25 fps. The claim is that these components are integrated in one modular system, so a GRETA-controlled agent perceives human users' behavior and adapts its own behavior accordingly, including in the socially under-explored modality of touch.","pith_inferences":["Because the paper reports no end-to-end latency, a direct perception-to-render benchmark is the natural next test; the architecture's practical value depends on that measurement, not on any single module's speed.","The chunk-and-thread incremental realizer is a general pattern that any behavior realizer facing interruptions or repairs could adopt, even outside the GRETA system.","The social-touch framework suggests a transfer path to physical robots with tactile sensing, though duration and pressure sensing would need to be added for full parity.","The reinforcement-learning dialogue planner could become a testbed for comparing adaptation policies under identical perception and rendering conditions."],"forward_implications":["A GRETA agent can cut off or revise its own behavior mid-gesture, making interruptions, repairs, concurrent feedback, and fast turn-taking computationally reachable.","Frame-level rendering at 25 fps lets the agent's facial expressions and head or gaze behavior track the user's audio-visual behavior continuously rather than utterance by utterance.","The same perception-to-realization loop can be repurposed across scenarios by changing the content messages, so motivational interviewing, CBT, and social-skills training can share one platform.","In VR, the agent can recognize four types of human touch and modulate whether to touch back based on computed rapport and proxemics.","The modular design lets the frame-level behavior prediction model be swapped, so deeper or different adaptation models can be plugged into the same real-time loop."],"supporting_citations":[{"why":"Defines the three-layer architecture and markup languages that the GRETA pipeline extends.","marker":"[31]"},{"why":"Presents the original GRETA platform that this paper's extensions build on.","marker":"[41]"},{"why":"Supplies the Meaning Miner behavior planner that translates communicative intentions into gestures.","marker":"[24]"},{"why":"Supplies the rule-based nonverbal behavior generator used in the behavior planning layer.","marker":"[34]"},{"why":"Presents the integrated interactive adaptive system that realizes the real-time adaptive loop.","marker":"[60]"},{"why":"Provides the frame-level behavior prediction model that drives the agent's adaptation at each frame.","marker":"[61]"},{"why":"Provides facial feature extraction used to perceive the user's expressions, head movements, and gaze.","marker":"[3]"},{"why":"Provides audio feature extraction used to perceive the user's speech prosody.","marker":"[14]"},{"why":"Supplies the dialogue manager that controls conversation flow and turn-taking in the use cases.","marker":"[55]"},{"why":"Supplies the appraisal-based emotion model used by the intention planner, including for social touch decisions.","marker":"[11]"}],"fun_headline_variants":["Modular GRETA platform reads faces, speech, and touch in real time","Adaptive virtual agents get real-time social cue processing","GRETA platform adapts agents to faces, speech, and even touch","Platform lets virtual agents read faces and adapt behavior live"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the entire external-tool chain, including speech recognition, face and audio feature extraction, interpretation, behavior prediction, message passing, and rendering, completing within the stated 0.04 second loop, but the paper asserts the 25 fps rate without reporting end-to-end latency measurements.","fun_headline_variants_meta":{"raw":{"variants":["Modular GRETA platform reads faces, speech, and touch in real time","Adaptive virtual agents get real-time social cue processing","GRETA platform adapts agents to faces, speech, and even touch","Platform lets virtual agents read faces and adapt behavior live"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000448,"raw_usage":{"total_tokens":2196,"prompt_tokens":818,"completion_tokens":1378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":1306}},"tokens_in":434,"tokens_out":1378,"duration_ms":10132,"temperature":1.0,"reasoning_tokens":1306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:42:26.268761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument a running session and timestamp each hop; if the measured time from a user's facial or audio change to the agent's displayed frame systematically exceeds one 40 ms frame period, the platform does not deliver the real-time adaptation it claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the three-layer architecture and markup languages that the GRETA pipeline extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the original GRETA platform that this paper's extensions build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rule-based nonverbal behavior generator used in the behavior planning layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the integrated interactive adaptive system that realizes the real-time adaptive loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the frame-level behavior prediction model that drives the agent's adaptation at each frame."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides facial feature extraction used to perceive the user's expressions, head movements, and gaze."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dialogue manager that controls conversation flow and turn-taking in the use cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the appraisal-based emotion model used by the intention planner, including for social touch decisions."}],"review_version":1}