REVIEW 8 cited by
Body of Her: A Preliminary Study on End-to-End Humanoid Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Body of Her: A Preliminary Study on End-to-End Humanoid Agent
read the original abstract
Interactive virtual humanoid agent is a crucial interface with the physical world. A relatively complete humanoid agent first needs to have face and body, then possess both verbal and non-verbal (such as eye contact, facial expression, lip motion, gesture, and manipulation) abilities, and finally, it is capable of real-time duplex communication, e.g., the ability to actively interrupt conversations. Most prior systems typically only consider a subset of these elements, leaving a gap from realistic humanoid agent. In this work, we propose a real-time, duplex, interactive end-to-end network capable of modeling realistic agent behaviors, including speech, full-body movements for talking, responding, idling, and manipulation. This system is a multimodal model integrating audio and visual inputs, extended from a pre-trained large language model (LLM). We collect approximately 200,000 hours of audio, around 130,000 hours of video data, and about 20,000 alignment samples to build the model. The final model demonstrates capabilities that are difficult to achieve in previous systems, such as generalized object manipulation. This work performs a preliminary exploration of the end-to-end approach in this field, aiming to inspire further research towards scaling up.
Forward citations
Cited by 8 Pith papers
-
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Wan-Streamer is a unified end-to-end Transformer for low-latency streaming audio-visual interaction using block-causal attention on interleaved multimodal tokens.
-
Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
Live Avatar reports real-time streamable generation from a 14B audio-driven diffusion model at ~20 FPS on 5 H800s with stable identity over 10,000 seconds.
-
Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
Live Avatar enables 45 FPS real-time streaming infinite-length audio-driven avatar generation from a 14B diffusion model via distillation and timestep-forcing pipeline parallelism.
-
OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation
A single autoregressive diffusion model, trained on a new 286-hour SMPL-X dataset, generates whole-body motion from text, audio, and spatial-temporal control signals, with reference-motion conditioning.
-
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Wan-Streamer presents a unified end-to-end Transformer for low-latency multimodal streaming interaction without external modules.
-
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Wan-Streamer is a unified Transformer model for low-latency streaming audio-visual interaction that jointly handles perception, reasoning, generation, and timing without external modules.
-
Wan-Streamer v0.2: Higher Resolution, Same Latency
Wan-Streamer v0.2 upgrades native-streaming audio-visual interaction to 640×368 at 25 FPS with unchanged ~200 ms model-side latency via a single-GPU thinker and multi-GPU Ulysses-style performer.
-
Video = World + Event Stream
Wan-Streamer v0.3 reformulates streaming video as a persistent world plus a changing event stream and extends an agent's output with open-vocabulary behavior, without reporting any quantitative results.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.