Pith. sign in

REVIEW 5 major objections 5 minor 83 references

HelloWorld: Enabling Socially Interactive Characters in Video World Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read HelloWorld is a video world model whose characters can be prompted to wave, nod, or speak to the viewer at a chosen moment.

desk verdict First video world model with viewer-directed social interactions; the temporal mask is a clever idea and the benchmark is useful, but the evaluation needs error bars and a clearer CamCtrl protocol before I'd trust the camera-control numbers. read the letter →

arxiv 2608.05070 v1 pith:XHFOXYAO submitted 2026-08-05 cs.CV

classification cs.CV
keywords videoworldmodelssocialinteractioncameracontrolself-distillationtemporalattentionmaskdiffusiontransformercharacteranimationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces HelloWorld, a video world model whose on-screen characters can be prompted to socially interact with the viewer — turning toward the camera, waving, nodding, or saying a short greeting — at a user-specified moment with a single button press. The central claim is that a model can learn this interaction without losing camera control or video quality. The method uses self-distillation: the base model generates its own interaction-rich training clips, an off-the-shelf geometry tool recovers camera trajectories and point clouds from those clips, and the model is finetuned on the clips under a re-rendered warp-video condition that encodes the target camera motion. At inference, a training-free temporal cross-attention mask makes the interaction text prompt attend only to frames inside the press window, so the character behaves ambiently outside it. On a new 400-sample benchmark, HelloWorld reaches 81.7% interaction-timing accuracy versus roughly 30–52% for baselines and the lowest gaze deviation, while keeping the best camera-following score.

What carries the argument

The load-bearing piece is the warp-video condition: a pseudo-video that lifts the first frame into a point cloud and reprojects it along the target camera path, giving the diffusion transformer an explicit, frame-aligned geometric history condition after visible-token selection discards reprojection holes. The second carrier is the self-distillation loop, in which the frozen base model's own interaction-rich clips are annotated for camera trajectory and point cloud, then reconstructed by a LoRA-finetuned DiT under the flow-matching loss, with the camera-text prompt excluded so camera control is carried solely by the warp video. The third is the temporal cross-attention mask, a training-free modification of Eq. (4) that sets attention to $-\infty$ for query tokens outside the interaction window attending to interaction text tokens, thereby temporally localizing the character's response.

What would settle it

Run the same pose-recovery tool on clips rendered with a known ground-truth camera path, such as a synthetic 3D scene with an animated camera, and measure per-frame pose error; if the error is large on stylized or synthetic content, or if adding small noise to the recovered poses significantly degrades HelloWorld's CamCtrl and TimeAcc, then the self-distillation geometry pipeline is not doing the load-bearing work claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is that social interaction can be made a first-class input to a video world model, on par with camera trajectory and text prompt. HelloWorld takes a first-frame image, a text prompt, a camera trajectory, and an interaction window, and generates a video in which the character both follows the camera and performs the prompted viewer-directed action within the window. The method is self-distillation: the frozen base model LTX-2.3 synthesizes clips containing both interactions and camera motion; Pi3X recovers each clip's camera path and first-frame point cloud; a re-rendered warp video becomes the geometric condition; and a rank-32 LoRA is finetuned under flow matching, with the camera text prompt deliberately removed so camera control comes only from the warp video. At inference, a temporal cross-attention mask blocks frames outside the press window from attending to the interaction prompt, localizing the response in time. On HelloWorldBench, HelloWorld reports TimeAcc 81.7% versus 30.9–52.6% for the compared world models, GazeDev 40.2 degrees versus 52.8–77.2 degrees, and the highest CamCtrl at 82.9, while staying competitive on background consistency and aesthetic score; on ActAcc its 41.4% trails LingBot-World's 50.5%, which the paper reads as an untimed behavior that ActAcc rewards regardless of when the action occurs.

Load-bearing premise

The load-bearing premise is that the frozen base model generates videos that genuinely contain both social interactions and natural camera motion, and that the off-the-shelf geometry tool recovers accurate camera poses and first-frame point clouds from those specific generated videos; if pose recovery is wrong on synthetic content, the warp-video condition injects incorrect guidance and the claimed joint preservation of interaction quality and camera control collapses.

Editorial extensions

If this is right

  • Users can specify not only what a character does but when it does it, with interaction timing accuracy of 81.7% compared with roughly random 30–52% for trajectory-controlled baselines.
  • Camera-pose following does not have to be traded for interaction quality: HelloWorld keeps the best CamCtrl score (82.9) and competitive aesthetic and background-consistency scores among the compared models.
  • The system needs no manually collected or annotated interaction video: all training data are generated by the base model itself and processed automatically.
  • Training-free temporal control extends to audio: masking both video and audio cross-attention streams raises SpeechInWin from 52.5% to 69.1%, localizing spoken greetings to the interaction window.
  • HelloWorldBench introduces three metrics (ActAcc, TimeAcc, GazeDev) that separate what action occurs, when it occurs, and whether it is directed at the viewer, making future social-interaction world models measurable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The self-distillation recipe should transfer to other base video generators, since it only needs the base model to sample interaction-rich clips and an off-the-shelf geometry tool to annotate them; the cost is a single lightweight fine-tuning run per base model.
  • The temporal cross-attention mask is a general event-localization mechanism: the same $-\infty$ masking applied to interaction-related tokens could gate other user-specified events, such as object changes, weather effects, or sound cues, without retraining.
  • If pose recovery from generated videos is the bottleneck, a stronger or self-supervised geometry estimator should directly improve GazeDev and CamCtrl; the benchmark could also be extended with adversarial timing windows, such as sub-second presses or overlapping windows, to expose the limits of the mask.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HelloWorld, a video world model that lets users prompt an on-screen character to perform viewer-directed social interactions (waving, nodding, speaking, etc.) within a user-specified time window. The method has two components: a self-distillation finetuning stage in which the base LTX-2.3 model is finetuned on its own generated interaction-rich videos under a warp-video camera condition, and a training-free temporal cross-attention mask that suppresses interaction-related text tokens outside the button-press window. The authors also introduce HelloWorldBench, a 400-sample benchmark with three interaction metrics (ActAcc, TimeAcc, GazeDev) and three conventional metrics (BgCons, Aesthetic, CamCtrl). Experiments report large gains in TimeAcc (81.7% vs. roughly 30–52% for baselines) and GazeDev, competitive ActAcc, and strong user-study preferences.

Significance. If the claims hold, the paper makes a useful contribution: it is the first to address social interaction with the viewer as a first-class capability of video world models, and the temporal cross-attention mask is a clean, training-free mechanism for event timing. The benchmark is a substantial new resource, with 400 samples and interaction-specific metrics, and the user study provides evidence beyond automatic metrics. The main uncertainty is evaluative: the headline quantitative claims rest on metrics that are incompletely specified, one of which excludes failures by construction, and the central camera-control result depends on an unvalidated pose estimator used in the training loop.

major comments (5)
  1. [§5.3, Table 1] The claim in Section 5.3 that HelloWorld outperforms existing methods on all three interaction metrics is contradicted by the reported ActAcc values: HelloWorld scores 41.4, while LTX-2.3 scores 42.5 and LingBot-World scores 50.5. The text later acknowledges that LingBot-World attains the highest ActAcc, but the introductory and abstract-level claim is not qualified. Please restate the claim to make clear that ActAcc is not state-of-the-art, and discuss the trade-off between temporal localization and action accuracy, as the no-mask ablation in Table 3 already suggests.
  2. [§3.2, Eq. (1)–(3)] The training loop uses Pi3X to recover per-frame camera poses and first-frame point clouds from LTX-2.3's own synthetic generations, yet the paper provides no validation of Pi3X on that distribution. Since the warp video is the only camera-conditioning input, systematic pose or depth errors on interaction-rich, non-rigid content would propagate directly into the learned conditioning and could explain part of the reported camera-control performance. Please add a quantitative validation, e.g., comparing Pi3X-estimated trajectories on held-out self-generated clips against known synthetic trajectories or human-annotated correspondences, and report the resulting pose error.
  3. [§4.2, Table 1] CamCtrl is never defined in the manuscript. If it is computed with the same Pi3X estimator that generates the training conditions, the reported 82.9 could partly reflect estimator bias rather than true camera following. Please state the exact computation, identify the estimator and any alignment step, and, if possible, evaluate camera following with an independent measure (e.g., optical-flow consistency or a different pose estimator) to rule out circularity.
  4. [§4.2, TimeAcc definition] The TimeAcc computation excludes videos in which the VLM judge selects the 'no interaction' option. This can inflate the metric, because a model that never performs the interaction is never penalized for timing; it also makes the 81.7% figure not directly comparable across methods if they produce different shares of no-interaction outputs. Please report TimeAcc computed over all samples (treating 'no interaction' as incorrect) and report the per-method exclusion rates, or justify why the excluded cases are uninformative.
  5. [§5.2, Table 1] The paper states that each sample is run with three seeds and that averaged results are reported, but the tables contain no error bars, confidence intervals, or significance tests. Without these, several headline gaps (e.g., ActAcc 41.4 for HelloWorld vs. 42.5 for LTX-2.3, and differences among TimeAcc values near 30–40%) could be within run-to-run noise. Please report variance or confidence intervals for the main metrics and, where appropriate, a paired significance test across the three seeds.
minor comments (5)
  1. [§4.2, GazeDev] GazeDev is computed only on the 217 human samples, but the paper does not explain why 217 of the 400 samples are human or how the non-human samples contribute to the headline GazeDev number; please clarify the sample split and the aggregation.
  2. [§5.2, VLM judge] The evaluation uses 'Qwen3.6-35B-A3B' as the VLM judge; please provide a precise model identifier or checkpoint and include the full evaluation prompts used for ActAcc, TimeAcc, and GazeDev in an appendix to make the benchmark reproducible.
  3. [§4.2, §5.5] SpeechInWin and the user-study question 'Action Naturalness' are used without a formal definition in the main text; please define them explicitly, including how Whisper transcriptions are matched to the interaction window.
  4. [§5.4, Table 2] The 'Real-video' ablation setting is said to train on real data from Warp-as-History [60], but the number of real clips, their resolution, and the training steps are not stated; please report these details so that the comparison with the self-distillation data is interpretable.
  5. [§3.3, Fig. 1] The interaction button F is described as a press that opens a window [τs, τe], but Figure 1 and the text do not clarify whether a single click or a press-and-hold defines the window; please state how users specify both the start and the duration of the interaction.

Circularity Check

0 steps flagged · score 2.0 of 10

No demonstrated circularity: the only self-citation is corroborated in-paper, and the Pi3X/CamCtrl loop is an unspecified risk, not a shown reduction.

full rationale

Walking the claimed derivation chain, no step reduces a reported result to its own input by definition. (i) Self-distillation (Sec. 3.2) is a reconstruction objective: Eq. (3) trains the LoRA to predict the velocity field of the base model's own generated video given the warp-video condition derived from that same video; the reported interaction metrics are then measured on external Unsplash first frames (Sec. 4.1) with an external VLM judge and UniGaze, so the evaluation is not fitted to the training signal. (ii) The temporal gate of Eq. (4) directly suppresses cross-attention to interaction tokens outside the press window; high TimeAcc is therefore partly a check that the architectural mechanism operates as specified, and the ablation in Table 3 (no-mask TimeAcc 52.5 vs. masked 81.7) shows the measurement is not vacuous, but this is an architectural intervention rather than a fitted input renamed as prediction. (iii) The only load-bearing self-citation is [42] for 'The base video generation model itself is capable of producing rich social interactions'; Table 1 independently corroborates this with LTX-2.3's ActAcc of 42.5, so the argument does not reduce to a self-citation chain. (iv) The Pi3X loop is a legitimate correctness risk: if CamCtrl is scored by the same pose estimator used to build the warp-video training conditions, then camera-control numbers could be inflated; however, the paper never defines how CamCtrl is computed, so applying the hard rule that circularity must be exhibited by quoted equations, this remains an unverified risk rather than a demonstrated self-reduction. Overall the benchmark and the main interaction metrics are externally grounded; score 2 reflects only the minor, non-load-bearing self-citation and the unquantified estimator-alignment risk.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on four unverified domain assumptions: the base model's interaction ability, Pi3X's geometric accuracy, the transferability of warp conditioning, and the validity of VLM judgments. These are not fatal, but they mean the reported numbers should be read as initial evidence rather than settled measurements. The method itself introduces no new physical or conceptual entities; its new components are procedural (self-distillation, temporal mask, benchmark).

free parameters (4)
  • LoRA rank = 32
    Chosen by hand for the finetuning adapter; no sensitivity analysis is reported.
  • Learning rate = 1e-4
    Training hyperparameter for self-distillation; no ablation or grid search is shown.
  • Training steps = 2000
    Short finetune on 156 videos; no analysis of convergence or overfitting.
  • Warp attention strength = 0.3
    Blending weight for warp reference tokens; set by hand with no sweep reported.
assumptions (4)
  • domain assumption Base model LTX-2.3 reliably generates interaction-rich videos when prompted
    Required for self-distillation data; asserted in Sec. 3.2 with support from the authors' own SocialDirector [42].
  • domain assumption Pi3X recovers accurate first-frame point clouds and per-frame camera poses from generated videos
    Warp videos in Eq. (1) inherit any Pi3X error; no validation on synthetic content is provided.
  • domain assumption Warp-video conditioning from [60] works for LTX-2.3 with LoRA finetuning
    Adopted as the camera-control backbone without independent verification in this paper.
  • domain assumption Qwen3.6-35B-A3B VLM judge correctly identifies actions and timing
    ActAcc and TimeAcc depend entirely on this judge; no human agreement on the benchmark metrics is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HelloWorld: Enabling Socially Interactive Characters in Video World Models." pith.science (2026). https://pith.science/paper/XHFOXYAO

@misc{pith2026260805070,
  author       = {Pith},
  title        = {Pith review of: HelloWorld: Enabling Socially Interactive Characters in Video World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHFOXYAO}},
  note         = {Machine review of arXiv:2608.05070}
}
read the original abstract

Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 38 canonical work pages

  1. [1]

    Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

    Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai.arXiv preprint arXiv:2606.02800, 2026

  2. [2]

    From masks to worlds: A hitchhiker’s guide to world models.arXiv preprint arXiv:2510.20668, 2025

    Jinbin Bai, Yu Lei, Hecong Wu, Yuchen Zhu, Shufan Li, Yi Xin, Xiangtai Li, Molei Tao, Aditya Grover, and Ming-Hsuan Yang. From masks to worlds: A hitchhiker’s guide to world models.arXiv preprint arXiv:2510.20668, 2025

  3. [3]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

  4. [4]

    Philip J. Ball, Jakob Bauer, Frank Belletti, Bethanie Brownfield, Ariel Ephrat, Shlomi Fruchter, Agrim Gupta, Kristian Holsheimer, Aleksander Holynski, Jiri Hron, Christos Kaplanis, Marjorie Limont, Matt McGill, Yanko Oliveira, Jack Parker-Holder, Frank Perbet, Guy Scully, Jeremy Shar, Stephen Spencer, Omer Tov, Ruben Villegas, Emma Wang, Jessica Yung, Ci...

  5. [5]

    Ambigest: A dataset of social gestures with inter-class similarity and intra-class variability

    Hajra Anwar Beg, Mohamed Daoudi, and Angela Bartolo. Ambigest: A dataset of social gestures with inter-class similarity and intra-class variability. In2026 IEEE 20th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–5. IEEE, 2026

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/ video-generation-models-as-world-simulators

  7. [7]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

  8. [8]

    Socialgesture: Delving into multi-person gesture understanding

    Xu Cao, Pranav Virupaksha, Wenqi Jia, Bolin Lai, Fiona Ryan, Sangmin Lee, and James M Rehg. Socialgesture: Delving into multi-person gesture understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19509–19519, 2025

Show all 83 references
  1. [9]

    Gamegen-x: Interactive open- world game video generation

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open- world game video generation. InInternational Conference on Learning Representations, volume 2025, pages 37546–37593, 2025

  2. [10]

    Unils: End-to-end audio-driven avatars for unified listening and speaking

    Xuangeng Chu, Ruicong Liu, Yifei Huang, Yun Liu, Yichen Peng, and Bo Zheng. Unils: End-to-end audio-driven avatars for unified listening and speaking. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25142–25152, 2026

  3. [11]

    Oasis: A universe in a transformer.URL: https://oasis-model

    Etched Decart, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a transformer.URL: https://oasis-model. github. io, 2(3):6, 2024

  4. [12]

    Worldscore: A unified evaluation benchmark for world generation

    Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. Worldscore: A unified evaluation benchmark for world generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 27713–27724, 2025

  5. [13]

    The matrix: Infinite-horizon world generation with real-time moving control.Advances in Neural Information Processing Systems, 38:87318–87344, 2026

    Ruili Feng, Han Zhang, Zhilei Shu, Zhantao Yang, Longxiang Tang, Zhicai Wang, Andy Zheng, Jie Xiao, Zhiheng Liu, Ruihang Chu, et al. The matrix: Infinite-horizon world generation with real-time moving control.Advances in Neural Information Processing Systems, 38:87318–87344, 2026

  6. [14]

    Longvie 2: Multimodal controllable ultra-long video world model

    Jianxiong Gao, Zhaoxi Chen, Xian Liu, Junhao Zhuang, Chengming Xu, Jianfeng Feng, Yu Qiao, Yanwei Fu, Chenyang Si, and Ziwei Liu. Longvie 2: Multimodal controllable ultra-long video world model. arXiv preprint arXiv:2512.13604, 2025

  7. [15]

    Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025

    Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025

  8. [16]

    Ltx-2: Efficient joint audio-visual foundation model.arXiv preprint arXiv:2601.03233, 2026

    Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, et al. Ltx-2: Efficient joint audio-visual foundation model.arXiv preprint arXiv:2601.03233, 2026

  9. [17]

    Voyager: Long-range and world-consistent video diffusion for explorable 3d scene generation.ACM Transactions on Graphics (TOG), 44(6):1–15, 2025

    Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson Lau, Wangmeng Zuo, et al. Voyager: Long-range and world-consistent video diffusion for explorable 3d scene generation.ACM Transactions on Graphics (TOG), 44(6):1–15, 2025. 11

  10. [18]

    Bind-your-avatar: Multi- character-talking video generation with dynamic 3d-mask-based embedding router

    Yubo Huang, Weiqiang Wang, Sirui Zhao, Tong Xu, Lin Liu, and Enhong Chen. Bind-your-avatar: Multi- character-talking video generation with dynamic 3d-mask-based embedding router. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4440–4449, 2026

  11. [19]

    Can mllms read the room? a multimodal benchmark for assessing deception in multi-party social interactions.arXiv preprint arXiv:2511.16221, 2025

    Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu, and Yoichi Sato. Can mllms read the room? a multimodal benchmark for assessing deception in multi-party social interactions.arXiv preprint arXiv:2511.16221, 2025

  12. [20]

    Grasp: Learning to ground social reasoning in multi-person non-verbal interactions

    Junho Kim, Xu Cao, Houze Yang, Bikram Boote, Ana Jojic, Fiona Ryan, Bolin Lai, Sangmin Lee, and James M Rehg. Grasp: Learning to ground social reasoning in multi-person non-verbal interactions. arXiv preprint arXiv:2605.15764, 2026

  13. [21]

    Let them talk: Audio-driven multi-person conversational video generation.Advances in Neural Information Processing Systems, 38:70990–71013, 2026

    Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conversational video generation.Advances in Neural Information Processing Systems, 38:70990–71013, 2026

  14. [22]

    Modeling multimodal social interactions: new challenges and baselines with densely aligned representations

    Sangmin Lee, Bolin Lai, Fiona Ryan, Bikram Boote, and James M Rehg. Modeling multimodal social interactions: new challenges and baselines with densely aligned representations. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14585–14595, 2024

  15. [23]

    Worldmodelbench: Judging video generation models as world models.Advances in Neural Information Processing Systems, 38, 2026

    Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph Gonzalez, et al. Worldmodelbench: Judging video generation models as world models.Advances in Neural Information Processing Systems, 38, 2026

  16. [24]

    Hunyuan-gamecraft: High-dynamic interactive game video generation with hybrid history condition.arXiv preprint arXiv:2506.17201, 2(3):6, 2025

    Jiaqi Li, Junshu Tang, Zhiyong Xu, Longhuang Wu, Yuan Zhou, Shuai Shao, Tianbao Yu, Zhiguo Cao, and Qinglin Lu. Hunyuan-gamecraft: High-dynamic interactive game video generation with hybrid history condition.arXiv preprint arXiv:2506.17201, 2(3):6, 2025

  17. [25]

    Towards online multi-modal social interaction understanding.Transactions on Machine Learning Research (TMLR), 2026

    Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M Rehg, and Yapeng Tian. Towards online multi-modal social interaction understanding.Transactions on Machine Learning Research (TMLR), 2026

  18. [26]

    Omni-mmsi: Toward identity-attributed social interaction understanding

    Xinpeng Li, Bolin Lai, Hardy Chen, Shijian Deng, Cihang Xie, Yuyin Zhou, James M Rehg, and Yapeng Tian. Omni-mmsi: Toward identity-attributed social interaction understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684–8696, 2026

  19. [27]

    Sekai: A video dataset towards world exploration

    Zhen Li, Chuanhao Li, Xiaofeng Mao, Shaoheng Lin, Ming Li, Shitian Zhao, Zhaopan Xu, Xinyue Li, Yukang Feng, Jianwen Sun, Zizhen Li, Fanrui Zhang, Jiaxin Ai, Zhixiang Wang, Yuwei Wu, Tong He, Yunde Jia, and Kaipeng Zhang. Sekai: A video dataset towards world exploration. In D....

  20. [28]

    Wildworld: A large-scale dataset for dynamic world modeling with actions and explicit state toward generative arpg.arXiv preprint arXiv:2603.23497, 2026

    Zhen Li, Zian Meng, Shuwei Shi, Wenshuo Peng, Yuwei Wu, Bo Zheng, Chuanhao Li, and Kaipeng Zhang. Wildworld: A large-scale dataset for dynamic world modeling with actions and explicit state toward generative arpg.arXiv preprint arXiv:2603.23497, 2026

  21. [29]

    Intergen: Diffusion-based multi- human motion generation under complex interactions.International Journal of Computer Vision, 132(9): 3463–3483, 2024

    Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi- human motion generation under complex interactions.International Journal of Computer Vision, 132(9): 3463–3483, 2024

  22. [30]

    Polyslgen: Online multimodal speaking-listening reaction generation in polyadic interaction

    Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, and Xucong Zhang. Polyslgen: Online multimodal speaking-listening reaction generation in polyadic interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 29379–29390, 2026

  23. [31]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

  24. [32]

    Uvagaze: Unsupervised 1-to-2 views adaptation for gaze estimation

    Ruicong Liu and Feng Lu. Uvagaze: Unsupervised 1-to-2 views adaptation for gaze estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3693–3701, 2024. 12

  25. [33]

    Pnp-ga+: Plug-and-play domain adaptation for gaze estimation using model variants.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 (5):3707–3721, 2024

    Ruicong Liu, Yunfei Liu, Haofei Wang, and Feng Lu. Pnp-ga+: Plug-and-play domain adaptation for gaze estimation using model variants.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 (5):3707–3721, 2024

  26. [34]

    Single-to-dual-view adaptation for egocentric 3d hand pose estimation

    Ruicong Liu, Takehiko Ohkawa, Mingfang Zhang, and Yoichi Sato. Single-to-dual-view adaptation for egocentric 3d hand pose estimation. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 677–686. IEEE, 2024

  27. [35]

    Sfhand: A streaming framework for language-guided 3d hand forecasting and embodied manipulation.arXiv e-prints, pages arXiv–2511, 2025

    Ruicong Liu, Yifei Huang, Liangyang Ouyang, Caixin Kang, and Yoichi Sato. Sfhand: A streaming framework for language-guided 3d hand forecasting and embodied manipulation.arXiv e-prints, pages arXiv–2511, 2025

  28. [36]

    Generalizing gaze estimation with outlier-guided collaborative adaptation

    Yunfei Liu, Ruicong Liu, Haofei Wang, and Feng Lu. Generalizing gaze estimation with outlier-guided collaborative adaptation. InProceedings of the IEEE/CVF international conference on computer vision, pages 3835–3844, 2021

  29. [37]

    Playmate2: Training-free multi-character audio-driven animation via diffusion transformer with reward feedback.arXiv preprint arXiv:2510.12089, 2025

    Xingpei Ma, Shenneng Huang, Jiaran Cai, Yuansheng Guan, Shen Zheng, Hanfeng Zhao, Qiang Zhang, and Shunsi Zhang. Playmate2: Training-free multi-character audio-driven animation via diffusion transformer with reward feedback.arXiv preprint arXiv:2510.12089, 2025

  30. [38]

    Xiaofeng Mao, Zhen Li, Chuanhao Li, Xiaojie Xu, Kaining Ying, and Kaipeng Zhang. Yume1. 5: A text-controlled interactive world generation model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7752–7761, 2026

  31. [39]

    Multimediate: Multi-modal group behaviour analysis for artificial mediation

    Philipp Müller, Michael Dietz, Dominik Schiller, Dominike Thomas, Guanhua Zhang, Patrick Gebhard, Elisabeth André, and Andreas Bulling. Multimediate: Multi-modal group behaviour analysis for artificial mediation. InProceedings of the 29th ACM International Conference on Multim...

  32. [40]

    Leadership assessment in pediatric intensive care unit team training.arXiv preprint arXiv:2505.24389, 2025

    Liangyang Ouyang, Yuki Sakai, Ryosuke Furuta, Hisataka Nozawa, Hikoro Matsui, and Yoichi Sato. Leadership assessment in pediatric intensive care unit team training.arXiv preprint arXiv:2505.24389, 2025

  33. [41]

    Multi-speaker attention alignment for multimodal social interaction

    Liangyang Ouyang, Yifei Huang, Mingfang Zhang, Caixin Kang, Ryosuke Furuta, and Yoichi Sato. Multi-speaker attention alignment for multimodal social interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24608–24619, 2026

  34. [42]

    Socialdirector: Training- free social interaction control for multi-person video generation.arXiv preprint arXiv:2605.10079, 2026

    Liangyang Ouyang, Ruicong Liu, Caixin Kang, Yifei Huang, and Yoichi Sato. Socialdirector: Training- free social interaction control for multi-person video generation.arXiv preprint arXiv:2605.10079, 2026

  35. [43]

    Svbench: Evaluation of video generation models on social reasoning

    Wenshuo Peng, Gongxuan Wang, Tianmeng Yang, Chuanhao Li, Xiaojie Xu, Hui He, and Kaipeng Zhang. Svbench: Evaluation of video generation models on social reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 32872–32881, 2026

  36. [44]

    Dyadit: A multi-modal diffusion transformer for socially favorable dyadic gesture generation.arXiv preprint arXiv:2602.23165, 2026

    Yichen Peng, Jyun-Ting Song, Siyeol Jung, Ruofan Liu, Haiyang Liu, Xuangeng Chu, Ruicong Liu, Erwin Wu, Hideki Koike, and Kris Kitani. Dyadit: A multi-modal diffusion transformer for socially favorable dyadic gesture generation.arXiv preprint arXiv:2602.23165, 2026

  37. [45]

    Actavatar: Temporally-aware precise action control for talking avatars

    Ziqiao Peng, Yi Chen, Yifeng Ma, Guozhen Zhang, Zhiyao Sun, Zixiang Zhou, Youliang Zhang, Zhengguang Zhou, Zhaoxin Fan, Hongyan Liu, et al. Actavatar: Temporally-aware precise action control for talking avatars. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

  38. [46]

    Unigaze: Towards universal gaze estimation via large-scale pre-training

    Jiawei Qin, Xucong Zhang, and Yusuke Sugano. Unigaze: Towards universal gaze estimation via large-scale pre-training. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5809–5820, 2026

  39. [47]

    Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026

    Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URL https: //qwen.ai/blog?id=qwen3.6-35b-a3b. 13

  40. [48]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

  41. [49]

    Interact2ar: Full-body human-human interaction generation via autoregressive diffusion models

    Pablo Ruiz-Ponce, Sergio Escalera, José García-Rodríguez, Jiankang Deng, and Rolandos Alexandros Potamias. Interact2ar: Full-body human-human interaction generation via autoregressive diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

  42. [50]

    Solaris: Building a multiplayer video world model in minecraft.arXiv preprint arXiv:2602.22208, 2026

    Georgy Savva, Oscar Michel, Daohan Lu, Suppakit Waiwitlikhit, Timothy Meehan, Dhairya Mishra, Srivats Poddar, Jack Lu, and Saining Xie. Solaris: Building a multiplayer video world model in minecraft.arXiv preprint arXiv:2602.22208, 2026

  43. [51]

    Prisma-world: Camera-controllable multi-agent video world model

    Huiqiang Sun, Zhan Peng, Size Wu, Kun Wang, Kang Liao, Dianyi Wang, Xingyu Zeng, Sheng Jin, Yangguang Li, Zhiguo Cao, et al. Prisma-world: Camera-controllable multi-agent video world model. arXiv preprint arXiv:2606.09507, 2026

  44. [52]

    Worldplay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

    Wenqiang Sun, Haiyu Zhang, Haoyuan Wang, Junta Wu, Zehan Wang, Zhenwei Wang, Yunhong Wang, Jun Zhang, Tengfei Wang, and Chunchao Guo. Worldplay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

  45. [53]

    Alayaworld: Long-horizon and playable video world generation,

    AlayaWorld Team, Kaipeng Zhang, Chuanhao Li, Yifan Zhan, Yongtao Ge, Yuanyang Yin, Jiaming Tan, Kang He, Liaoyuan Fan, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, and Zihui Gao. Alayaworld: Long-horizon and playable video world generation,

  46. [54]

    Advancing open-source world models.arXiv preprint arXiv:2601.20540, 2026

    Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, et al. Advancing open-source world models.arXiv preprint arXiv:2601.20540, 2026

  47. [55]

    Unsplash: Beautiful, free images and photos

    Unsplash. Unsplash: Beautiful, free images and photos. https://unsplash.com, 2026. Accessed: 2026-07-17

  48. [56]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. InInternational Conference on Learning Representations, volume 2025, pages 73754–73776, 2025

  49. [57]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

  50. [58]

    Spatialvid: A large-scale video dataset with spatial annotations

    Jiahao Wang, Yufeng Yuan, Rujie Zheng, Youtian Lin, Jian Gao, Lin-Zhuo Chen, Yajie Bao, Chang Zeng, Yanxi Zhou, Xiao-Xiao Long, et al. Spatialvid: A large-scale video dataset with spatial annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

  51. [59]

    Fantasyportrait: Enhancing multi-character portrait animation with expression-augmented diffusion transformers

    Qiang Wang, Mengchao Wang, Fan Jiang, Yaqi Fan, Yonggang Qi, and Mu Xu. Fantasyportrait: Enhancing multi-character portrait animation with expression-augmented diffusion transformers. arXiv preprint arXiv:2507.12956, 2025

  52. [60]

    Warp-as-history: Generalizable camera-controlled video generation from one training video.arXiv preprint arXiv:2605.15182, 2026

    Yifan Wang and Tong He. Warp-as-history: Generalizable camera-controlled video generation from one training video.arXiv preprint arXiv:2605.15182, 2026

  53. [61]

    π3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025

    Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025

  54. [62]

    Reactivegwm: Steering npc in reactive game world models.arXiv preprint arXiv:2605.15256, 2026

    Zeqing Wang, Danze Chen, Zhaohu Xing, Zizhao Tong, Yinhan Zhang, Xingyi Yang, and Yeying Jin. Reactivegwm: Steering npc in reactive game world models.arXiv preprint arXiv:2605.15256, 2026

  55. [63]

    Interacthuman: Multi-concept human animation with layout-aligned audio conditions.arXiv preprint arXiv:2506.09984, 2025

    Zhenzhi Wang, Jiaqi Yang, Jianwen Jiang, Chao Liang, Gaojie Lin, Zerong Zheng, Ceyuan Yang, Yuan Zhang, Mingyuan Gao, and Dahua Lin. Interacthuman: Multi-concept human animation with layout-aligned audio conditions.arXiv preprint arXiv:2506.09984, 2025. 14

  56. [64]

    Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory.arXiv preprint arXiv:2604.08995, 2026

    Zile Wang, Zexiang Liu, Jiaxing Li, Kaichen Huang, Baixin Xu, Fei Kang, Mengyin An, Peiyu Wang, Biao Jiang, Yichen Wei, et al. Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory.arXiv preprint arXiv:2604.08995, 2026

  57. [65]

    Actworld: From explorable to interactive world model via action-aware memory.arXiv preprint arXiv:2606.17730, 2026

    Zhexiao Xiong, Yizhi Song, Hao Kang, Qing Yan, Liming Jiang, Jenson Yang, Zhoujie Fu, Stathi Fotiadis, Angtian Wang, Zichuan Liu, et al. Actworld: From explorable to interactive world model via action-aware memory.arXiv preprint arXiv:2606.17730, 2026

  58. [66]

    Inter-x: Towards versatile human-human interaction analysis

    Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile human-human interaction analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag...

  59. [67]

    Worldmark: A unified benchmark suite for interactive video world models.arXiv preprint arXiv:2604.21686, 2026

    Xiaojie Xu, Zhengyuan Lin, Kang He, Yukang Feng, Xiaofeng Mao, Yuanyang Yin, Kaipeng Zhang, and Yongtao Ge. Worldmark: A unified benchmark suite for interactive video world models.arXiv preprint arXiv:2604.21686, 2026

  60. [68]

    Holo-world: Unified camera, object and weather control for video world model.arXiv preprint arXiv:2606.20083, 2026

    Xiangchen Yin, Wenzhang Sun, Jiahui Yuan, Zijie Liu, Yinda Chen, Wei Li, Dachun Kai, Chunfeng Wang, and Xiaoyan Sun. Holo-world: Unified camera, object and weather control for video world model.arXiv preprint arXiv:2606.20083, 2026

  61. [69]

    Wbench: A comprehensive multi-turn benchmark for interactive video world model evaluation.arXiv preprint arXiv:2605.25874, 2026

    Kaining Ying, Hengrui Hu, Siyu Ren, Jiamu Li, Fengjiao Chen, Ziwen Wang, Xuezhi Cao, Xunliang Cai, and Henghui Ding. Wbench: A comprehensive multi-turn benchmark for interactive video world model evaluation.arXiv preprint arXiv:2605.25874, 2026

  62. [70]

    Building social world models with large language models.arXiv preprint arXiv:2606.11482, 2026

    Haofei Yu, Yining Zhao, Guanyu Lin, and Jiaxuan You. Building social world models with large language models.arXiv preprint arXiv:2606.11482, 2026

  63. [71]

    Socialgen: Modeling multi-human social interaction with language models

    Heng Yu, Juze Zhang, Changan Chen, Tiange Xiang, Yusu Fang, Juan Carlos Niebles, and Ehsan Adeli. Socialgen: Modeling multi-human social interaction with language models. In2026 International Conference on 3D Vision (3DV), pages 1–17. IEEE, 2026

  64. [72]

    Gamefactory: Creating new games with generative interactive videos

    Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11590–11599, 2025

  65. [73]

    Helios: Real real-time long video generation model.arXiv preprint arXiv:2603.04379, 2026

    Shenghai Yuan, Yuanyang Yin, Zongjian Li, Xinwei Huang, Xiao Yang, and Li Yuan. Helios: Real real-time long video generation model.arXiv preprint arXiv:2603.04379, 2026

  66. [74]

    Social-iq: A question answering benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817, 2019

  67. [75]

    Socioverse: A world model for social simulation powered by llm agents and a pool of 10 million real-world users.arXiv preprint arXiv:2504.10157, 2025

    Xinnong Zhang, Jiayu Lin, Xinyi Mou, Shiyue Yang, Xiawei Liu, Libo Sun, Hanjia Lyu, Yihang Yang, Weihong Qi, Yue Chen, et al. Socioverse: A world model for social simulation powered by llm agents and a pool of 10 million real-world users.arXiv preprint arXiv:2504.10157, 2025

  68. [76]

    Anytalker: Scaling multi-person talking video generation with interactivity refinement.arXiv preprint arXiv:2511.23475, 2025

    Zhizhou Zhong, Yicheng Ji, Zhe Kong, Yiying Liu, Jiarui Wang, Jiasun Feng, Lupeng Liu, Xiangyi Wang, Yanjia Li, Yuqing She, et al. Anytalker: Scaling multi-person talking video generation with interactivity refinement.arXiv preprint arXiv:2511.23475, 2025

  69. [77]

    Stereo magnification: learning view synthesis using multiplane images.ACM Transactions on Graphics (TOG), 37(4):1–12, 2018

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images.ACM Transactions on Graphics (TOG), 37(4):1–12, 2018

  70. [78]

    Social world models.arXiv preprint arXiv:2509.00559, 2025

    Xuhui Zhou, Jiarui Liu, Akhila Yerukola, Hyunwoo Kim, and Maarten Sap. Social world models.arXiv preprint arXiv:2509.00559, 2025

  71. [79]

    Omniworld: A multi-domain and multi-modal dataset for 4d world modeling.arXiv preprint arXiv:2509.12201, 2025

    Yang Zhou, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Haoyu Guo, Zizun Li, Kaijing Ma, Xinyue Li, Yating Wang, Haoyi Zhu, et al. Omniworld: A multi-domain and multi-modal dataset for 4d world modeling.arXiv preprint arXiv:2509.12201, 2025

  72. [80]

    Evaltalker: Learning to evaluate real-portrait-driven multi-subject talking humans.arXiv preprint arXiv:2512.01340, 2025

    Yingjie Zhou, Xilei Zhu, Siyu Ren, Ziyi Zhao, Ziwen Wang, Farong Wen, Yu Zhou, Jiezhang Cao, Xiongkuo Min, Fengjiao Chen, et al. Evaltalker: Learning to evaluate real-portrait-driven multi-subject talking humans.arXiv preprint arXiv:2512.01340, 2025. 15

  73. [81]

    Sana-wm: Efficient minute-scale world modeling with hybrid linear diffusion transformer

    Haoyi Zhu, Haozhe Liu, Yuyang Zhao, Tian Ye, Junsong Chen, Jincheng Yu, Tong He, Song Han, and Enze Xie. Sana-wm: Efficient minute-scale world modeling with hybrid linear diffusion transformer. arXiv preprint arXiv:2605.15178, 2026

  74. [82]

    Astra: General interactive world model with autoregressive denoising.arXiv preprint arXiv:2512.08931, 2025

    Yixuan Zhu, Jiaqi Feng, Wenzhao Zheng, Yuan Gao, Xin Tao, Pengfei Wan, Jie Zhou, and Jiwen Lu. Astra: General interactive world model with autoregressive denoising.arXiv preprint arXiv:2512.08931, 2025. 16

  75. [2026]

    URLhttps://arxiv.org/abs/2607.06291

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.