{"id":"2036d064-b601-46af-abe5-02db4d2eeab4","arxiv_id":"2607.23517","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A distilled video model jointly controls multi-scale upper-body motion latents and two discrete contact states to generate real-time human–object interaction at 25 FPS.","lead":"The paper builds a real-time video generator that drives a person's upper body, hands, and face from live motion while switching object contact with short text commands like \"grasp\" or \"no contact.\" It targets telepresence and interactive agents that need both fine motion control and contact-aware local scene changes at interactive rates.","discovery_kind":"new_method","skeptic_critique":null,"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript presents a video generation system for upper-body human–object interaction framed as a \"human-centric local world model.\" Control is factored into (i) a continuous human-state latent built by multi-scale motion encoding — separate ViT encoders for upper body, left/right hand (shared weights), and face regions, fused and implicitly retargeted to the reference identity via an attention decoder (§3.2, Eqs. 3–4) — and (ii) a discrete interaction state s_o ∈ {no contact, grasp} encoded as fixed T5 text phrases (§3.3). Interaction supervision comes from a fully synthetic pipeline (Z-Image references, Qwen-VL object crops, Qwen3 prompts, Wan2.2-14B-I2V videos, VL filtering; ~30K labeled samples). Training is two-stage (§3.5): human-state pretraining on >3M upper-body videos with flow matching + RePA + keypoint auxiliary losses, then interaction learning with the motion encoder frozen. The model is distilled via CausVid + Self-Forcing to 4-step causal sampling, claiming 25 FPS streaming at 512×512 with ~1000 ms latency on two H100s. Experiments (Tab. 1) compare against Wan-Animate, Wan2.2-TI2V-5B, Humo, MAI, VACE on FVD/tLPIPS/CLIP-I/OQ/HandSIM/FaceSIM/IdentitySIM plus a user study.","tokens_in":20840,"tokens_out":8939,"duration_ms":328793,"significance":"If the claims hold, this is a useful systems contribution: streaming human-driven interaction generation at 25 FPS with ~1 s latency and explicit, switchable contact control is a practical step toward interactive avatars and telepresence, and most prior work offers either motion control or text prompting, not both. Specific strengths worth crediting: the discrete-state-as-command design is simple, scene-agnostic, and directly falsifiable via a compliance rate; the multi-scale implicit latent with a shared hand encoder and implicit retargeting is a coherent extension of prior latent-motion work; training is unusually well specified (λ_rep=0.1, λ_kp=10, stage step counts, GPU counts, 45-frame clips); and the promised release of the 30K state-labeled interaction dataset would benefit the community. The \"world model\" framing, however, currently exceeds what a two-state contact controller demonstrates.","major_comments":[{"comment":"The 100-clip interaction test set is produced by the same generative pipeline (Z-Image references, Qwen-VL crops, Qwen3 prompts, Wan2.2-14B-I2V, VL filtering) that created the Stage-II training data. Disjoint identities/objects/prompts do not make this an independent sample: the FVD/tLPIPS/CLIP-I/OQ ground truth is Wan2.2-14B output, i.e., the teacher distribution the model was trained to match, while the baselines are not so matched. Gains may reflect distribution matching rather than more realistic interaction. Please add at least one off-pipeline evaluation: real upper-body HOI footage, held-out clips from a different video generator, or human/VLM contact-plausibility ratings on real reference images.","section":"§4.2, Tab. 1"},{"comment":"(i) HandSIM/FaceSIM are reported only against Wan-Animate ('–' for all other baselines); if these require motion-GT alignment that text-driven baselines cannot satisfy, state this and consider per-task tables. (ii) Each baseline's conditioning (driving video? object image? prompt source?) is unspecified, so the user-study axes TF/AF are not interpretable. (iii) The text claims 'favorable performance,' but MAI beats Ours on FVD (152.7 vs 154.1), tLPIPS (0.176 vs 0.178), and VQ (3.91 vs 3.82); this should be acknowledged rather than elided. (iv) The user study reports no rater count, clip count, protocol (Likert vs pairwise), or significance testing.","section":"§4.1, Tab. 1"},{"comment":"The discrete interaction state is the paper's central control contribution, yet its effect is shown only qualitatively in Fig. 8. A state-compliance rate (commanded s_o vs realized contact, using the authors' own Qwen-VL filter or human raters) over the test set would directly test the claim and is cheap to compute. Mid-stream state switching — the motivating real-time use case — is unquantified (frames-to-effect, failure modes). The 'grasp-by-default' bias of the w/o-text variant should also be quantified; the label balance of the 30K dataset is not reported.","section":"§4.4, Fig. 8"},{"comment":"The manuscript never states whether Tab. 1 reports the 20-step teacher or the 4-step distilled student. Since the headline claim couples real-time streaming with improved fidelity, the distillation cost must be measured: report FVD/tLPIPS/HandSIM before and after CausVid + Self-Forcing, plus resolution, sliding-window size, and how the two H100s are used (single stream vs sharded). If Tab. 1 is the teacher, the deployed system's quality at 25 FPS is currently unverified.","section":"§3.4, Tab. 1"},{"comment":"The 100 reenactment driving videos come from SpeakerVid-5M, which also supplies Stage-I training clips, and no decontamination statement is given. If any test video or near-duplicate appears in training, the HandSIM/FaceSIM/IdentitySIM gains over Wan-Animate partly reflect memorization. Please confirm exclusion or re-evaluate on a disjoint motion source.","section":"§4.2"}],"minor_comments":[{"comment":"Title/abstract/§5: the 'world model' framing is stronger than what is demonstrated — the interaction state has cardinality 2, and there is no persistence, memory, or scene-state evolution beyond contact switching. Either demonstrate additional scene states (e.g., object displacement persisting after release) or temper the framing to 'controllable interaction generation.'","section":"Title, §1, §5"},{"comment":"Second paragraph ends with a dangling citation list ('...contact and object manipulation. [Bhatnagar et al. 2022; ...]') not attached to any grammatical claim.","section":"§1"},{"comment":"HandSIM/FaceSIM are defined as per-frame MSE distances to tracked coefficients but named 'SIM' (similarity) and tabulated with ↓; rename (e.g., HandMSE) or define the direction explicitly.","section":"§4.1, Tab. 1"},{"comment":"OQ is a custom composite (product of object dynamics score and object DINO consistency). Justify the product form, report the two components separately, and give the metric's range; absolute differences (0.147 vs 0.151) are small and no variance is given.","section":"§4.1"},{"comment":"L_kp 'predicts target 2D keypoints from the implicit motion variable' — specify which tensor is decoded (z_b/z_lh/z_rh/z_f or s_h), which keypoints (body/hand/face), and which detector supplies supervision.","section":"§3.5"},{"comment":"Give a breakdown of the ~1000 ms end-to-end latency (motion encoding, 4 DiT steps, VAE decode) and distinguish first-frame from steady-state latency; also reconcile '25 FPS on two H100 GPUs' (abstract/§3.4) with 'on H100 GPUs' (contributions).","section":"§3.4, Fig. 1"},{"comment":"Streaming demos use Gemini-generated references while the training pipeline uses Z-Image; comment on robustness to reference-image source, since references in-the-wild will differ from both.","section":"Fig. 10 vs §3.3"},{"comment":"State the shape/token layout of s_h as injected into the DiT (Fig. 2 shows in-context tokens; Eq. 4 gives a spatial feature map), and how noised target tokens attend to it.","section":"§3.2, Eq. (4), Fig. 2"},{"comment":"Typographical: 'changes, Overall' (comma splice, §1); 'Discrete Interaction State' heading lacks a period (§4.4); inconsistent 'Fig.7'/'Fig. 8' spacing; abstract rendering of the emphasized states lost spacing ('asno contactorgrasp').","section":"passim"},{"comment":"The dataset release promise (§3.3) is welcome; please also state whether code, the distilled model, and the evaluation splits will be released, and report the 30K dataset's grasp/no-contact label balance.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The strongest motion-driven baseline, Wan-Animate (Cheng et al. 2025), shares four authors with this manuscript (Ji, Qi, P. Zhang, B. Zhang), and much of the compared ecosystem (Wan2.2, VACE) comes from the same organization. I see no impropriety, but the evaluation would be materially strengthened by one external motion-driven baseline and by clarifying whether baseline numbers were run in-house under matched conditions. Separately, the comparative-realism claim currently rests entirely on evaluation within the team's own synthetic distribution; the requested off-pipeline evaluation is the item I would treat as a condition for acceptance. The promised dataset release would add real value and should be verified before publication."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a competent systems paper that packages multi-scale implicit motion control with two fixed contact tokens and ships a distilled 25 FPS streamer. It is useful engineering for telepresence-style agents, not a new theory of world models.\n\nWhat is actually new is the control split. Human motion is a fused multi-scale latent (body, hands, face) without explicit skeleton retargeting; contact is not free-form text but a tiny discrete command set (“no contact” / “grasp”) encoded by T5. They also built a ~30K synthetic HOI factory (Z-Image → Qwen crops/prompts → Wan I2V → filter) to supervise those labels, then CausVid + Self-Forcing distillation on Wan2.2-5B. Relative to pure reenactment (Wan-Animate) and prompt-only HOI video (Humo, VACE, MAI), that joint scheme is a real design choice, and the ablations in Figs. 7–8 back the two pieces: multi-scale helps hands/face; without the text state the model defaults to grasp.\n\nWhat it does well: method writeup is coherent (Eqs. 1–4, two-stage train with frozen motion encoder in stage II), HandSIM/FaceSIM/IdentitySIM beat Wan-Animate, user study favors text/action following, and they show live streaming with state switches. Citations cover the right motion, HOI, and efficiency lines.\n\nSoft spots, in proportion: (1) “Human-centric world model” is marketing for controllable local upper-body video; scene-state change is basically object contact under two labels. (2) Interaction eval is 100 held-out clips from the same generative pipeline as training—so coordination claims stay partly in-distribution. No real HOI video benchmark, no contact IoU/penetration metric. (3) Only two states; “grasp” is coarse. (4) 25 FPS on two H100s at 512² is real-time for a lab demo, not a light deploy. Free parameters (loss weights, steps, bottleneck dim) are normal for this genre.\n\nWho it is for: people building interactive avatars and real-time human video control. Not for robotics HOI geometry or general physical world models. Math is standard DiT/flow-matching; data story is the weak link, not the equations.\n\nI would send it to peer review at a solid CV systems venue. Ask for real-data HOI numbers, contact metrics, tempered framing, and data/code release. Worth engaging if you work on controllable human video; skip if you wanted a genuine world model.","headline":"Solid real-time upper-body HOI video system with a clean continuous/discrete control split; the “world model” label and same-pipeline synthetic eval are the main overreaches.","tokens_in":21510,"tokens_out":660,"would_cite":false,"duration_ms":21844,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Joint continuous multi-scale human motion control and discrete contact commands can generate real-time upper-body videos where a person both moves and controllably changes nearby object states.","keywords":["video generation","world model","human-object interaction","upper-body animation","multi-scale motion latent","discrete interaction state","real-time streaming"],"falsifier":"Run the same driving motion and object reference on held-out real (not pipeline-synthesized) upper-body videos with both grasp and no-contact labels: if hand–object contact, object identity, and user-controllable state switches collapse relative to the paper’s synthetic test set, the central claim does not hold outside that generative distribution.","tokens_in":21664,"feed_emoji":"🤝","tokens_out":938,"duration_ms":23906,"temperature":0.7,"pith_summary":"Most human video generators either animate a person in isolation or let free-form text vaguely describe interaction, so motion control, contact outcomes, and real-time use stay poorly joined. This paper argues that upper-body interactive generation is better cast as a small human-centric local world: continuous human state plus a discrete interaction state jointly determine the next frames. Human state is a single implicit latent built by encoding upper-body, hand, and face regions at multiple scales and fusing them without skeleton retargeting; interaction state is a short language command such as “no contact” or “grasp,” not an open prompt. Supervised on a dedicated synthetic interaction pipeline and distilled for streaming, the system runs at 25 FPS and reports better fine-grained motion fidelity and hand–object coordination than text-only or animation baselines. A sympathetic reader cares because this is a concrete path from pure motion replay toward interactive agents and telepresence that can actually pick things up or leave them alone on command.","feed_headline":"Stream graspable upper-body video at 25 FPS","feed_subtitle":"Multi-scale motion latents plus no-contact/grasp commands control hand–object scene changes in real time.","key_machinery":"Continuous–discrete joint control: a multi-scale implicit human state s_h formed by fusing upper-body, left-hand, right-hand, and face motion latents with reference features, plus a discrete interaction state s_o drawn from a tiny language-encoded set {no contact, grasp}, both fed to the generator as ŷ = G(x_ref, x_obj, s_h, s_o).","core_discovery":"Person-centered upper-body scene evolution can be synthesized in real time by jointly conditioning a video generator on a unified multi-scale implicit human-state latent and a language-encoded discrete interaction-state command. The human state captures coordinated body, hand, and face dynamics without explicit retargeting; the interaction state switches contact relations such as no contact versus grasp under similar motion, so the model controls both how the person moves and how nearby objects change.","pith_inferences":["Extending the discrete state set beyond two labels (place, push, release) is the natural next stress test of whether language-as-command stays stable.","If real-video fine-tuning fails to match synthetic held-out scores, the method may be more a controllable renderer of generative data than a world model of physical contact.","The same continuous–discrete split could transfer to egocentric or full-body settings where viewpoint and locomotion also change scene state."],"forward_implications":["Interactive agents and telepresence can switch grasp versus no-contact online without rewriting free-form prompts.","Fine hand and face motion can be driven cross-identity from multi-scale latents without skeleton retargeting errors.","Streaming 25 FPS generation on two H100-class GPUs becomes a practical target for person-centered local world models.","A small closed set of interaction-state commands can replace entangled text for contact-aware video control."],"fun_headline_variants":["Real-time upper-body world model jointly steers motion and grasp states","Multi-scale human latents plus grasp commands yield 25 FPS HOI video","Person-centered scenes evolve via implicit motion and discrete contact","Unified body-hand-face latent with no-contact/grasp control at 25 FPS","Stream controllable upper-body hand-object dynamics without retargeting"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That training and testing mostly on videos and labels produced by the same synthetic rendering pipeline, with only two scene-agnostic contact labels, is enough to claim realistic controllable hand–object coordination and local scene-state changes in the wild.","fun_headline_variants_meta":{"raw":{"variants":["Real-time upper-body world model jointly steers motion and grasp states","Multi-scale human latents plus grasp commands yield 25 FPS HOI video","Person-centered scenes evolve via implicit motion and discrete contact","Unified body-hand-face latent with no-contact/grasp control at 25 FPS","Stream controllable upper-body hand-object dynamics without retargeting"]},"model":"grok-4.5","effort":"low","cost_usd":0.004518,"raw_usage":{"total_tokens":1376,"prompt_tokens":880,"num_sources_used":0,"completion_tokens":82,"cost_in_usd_ticks":45184000,"prompt_tokens_details":{"text_tokens":880,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":414,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":880,"tokens_out":82,"duration_ms":8443,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T20:24:23.750467+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same driving motion and object reference on held-out real (not pipeline-synthesized) upper-body videos with both grasp and no-contact labels: if hand–object contact, object identity, and user-controllable state switches collapse relative to the paper’s synthetic test set, the central claim does not hold outside that generative distribution.","supporting_citations":[],"review_version":1}