Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Interspatial Attention for Efficient 4D Human Video Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Interspatial attention, a 3D-to-2D cross-attention block, powers a diffusion transformer that reports top scores for 4D human video generation, including camera control and multi-character scenes.

desk verdict ISA is a genuinely new conditioning mechanism with supporting ablations; the SOTA claim is real but currently backed by an evaluation protocol that needs matching CFG and released test sets. read the letter →

arxiv 2505.15800 v2 pith:HYG73EFJ submitted 2025-05-21 cs.CV

Ruizhi Shao , Yinghao Xu , Yujun Shen , Ceyuan Yang , Yang Zheng , Changan Chen , Yebin Liu , Gordon Wetzstein This is my paper
classification cs.CV
keywords humanvideogenerationdiffusiontransformerinterspatialattentionSMPLvariationalautoencodercameracontrolmulti-personanimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single diffusion-transformer architecture can generate photorealistic, view-consistent videos of one or more digital humans while precisely following a specified body pose and camera path. The authors argue that existing human video generators fail either because their video autoencoders blur fast human motion or because they inject 3D body models only as 2D renderings, losing geometric cues needed for occlusions and multi-person scenes. Their fix is a new cross-attention block, interspatial attention (ISA), in which tokens sampled from the SMPL body mesh and tokens from video frames attend to one another in a shared camera-coordinate space, plus a custom video VAE trained with spatio-temporal augmentation and regularization. On their test sets, the resulting model reports consistently higher PSNR, SSIM, LPIPS, and FVD than five prior methods, with the largest gaps on camera-moving and masked-background videos. A fair reader would care because the result suggests that explicit 3D templates and scalable video diffusion can be combined without sacrificing either photorealism or controllability.

What carries the argument

The central object is the symmetric interspatial attention (ISA) block with its interspatial positional encoding (ISPE). A shallow MLP encodes sinusoidally positioned samples from the SMPL mesh surface into 3D tokens; video latents provide 2D tokens. ISPE projects both sets of coordinates into normalized device coordinates using the known camera modelview-projection matrix (with 2D tokens placed on a zero-depth plane), adds sinusoidal positional encodings, and then two cross-attention operations let each modality query the other in a shared geometric frame. This bidirectionality is what the paper calls an implicit rendering-reconstruction loop, and it is what carries the claimed gains in occlusion handling and multi-view consistency. The other load-bearing component is the video VAE, whose causal 3D convolutions, spatio-temporal augmentation, and image-decoding regularization are what keep fast human motion from degrading the latent space.

What would settle it

Render a multi-person test video twice with the released model, once with accurate per-person SMPL meshes and once with the mesh of one person deliberately swapped for another, and compare the two generations under FVD/PSNR; a large drop would confirm that SMPL accuracy is the load-bearing input.

Watch

Extended reading notes

Core claim

The central claim is that ISA-DiT performs 4D human video synthesis at a higher quality than existing pose- and camera-conditioned generators. Concretely, the paper reports that the model outperforms AnimateAnyone, Champ, MusePose, Animate-X, and Human4DiT on all reported metrics and all three test configurations, e.g., FVD 143.6 versus 388.2 for Human4DiT on the static-background video set. The authors attribute the improvement to two coupled designs. First, the custom video VAE compresses fast human dynamics with less blur and yields latent distributions that diffusion training learns more easily, assisted by random structured motion, dynamic speed adjustment, and an image-decoding regularization that counteracts a last-frame bias. Second, ISA builds explicit geometric correspondences by projecting 3D SMPL surface points and 2D latent pixels into a unified normalized-device-coordinate space and running symmetric cross-attention in both directions, so 3D pose features propagate into video while video appearance features propagate back onto the mesh. The paper further claims the mechanism supports multi-character animation, controllable camera trajectories, and background composition.

Load-bearing premise

The method collapses if the 3D body-model pose estimates it consumes are not accurate enough to define trustworthy correspondences across frames and views, which the paper itself flags as a failure source for multi-person occlusions.

Editorial extensions

If this is right

  • Camera control becomes a first-class input: the same model can synthesize multiple views of a pose sequence from a moving camera, effectively turning video generation into a dynamic view-synthesis system.
  • Multi-character scenes are handled by concatenating each person's SMPL-derived tokens into the same ISA block, preserving identity and inter-character occlusion for dancing, boxing, and similar interactions.
  • Background composition is switchable: the model can place a generated human into a supplied background video or a zero background by concatenating background latents with the main latents.
  • Because ISA only needs a parametric template, the same block works with FLAME for facial expression generation, suggesting the mechanism is not tied to whole-body SMPL.
  • The video VAE's regularization produces latents that train diffusion models faster and to lower loss than unregularized latents, so the VAE design is a component separable from the attention mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The symmetric rendering-reconstruction view suggests ISA could be ported to other articulated objects with available templates, such as animal skeletons or deformable hand models, for comparable gains in consistency.
  • Because both modalities are aligned in NDC space, ISA's correspondence quality may degrade with extreme camera poses and fast cuts; the paper's own failure cases on wide viewpoint changes hint at this limit, and a natural extension is a more expressive, non-linear positional encoding for large view differences.
  • The last-frame-bias fix decomposes spatio-temporal latents into per-frame sub-latents; this is a generalizable recipe for other video VAEs and could be tested independently on existing tokenizers built for open video generation.
  • The scaling experiments reported in the supplement, if they follow typical DiT trends, suggest that ISA-DiT's advantage over baselines may grow with model size, placing 4D human generation on the same scaling curve as general video models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a 4D human video generation system consisting of three main components: a custom video VAE with spatio-temporal augmentation and image-decoding regularization; a symmetric "interspatial attention" (ISA) block that lets 3D SMPL tokens and 2D video tokens attend to each other using sinusoidal positional encodings in NDC space; and an ISA-based diffusion transformer (ISA-DiT) with identity, camera (Plücker), and background conditioning. The authors report large quantitative gains over AnimateAnyone, Champ, MusePose, Animate-X, and Human4DiT on three self-curated test sets (Video, Camera, Mask), report comparable VBench numbers to larger image-to-video models, and provide ablations supporting ISA and the VAE regularization. The paper's own limitations section (Sec. 7) acknowledges that errors in SMPL occlusion estimation and extreme camera movement cause artifacts.

Significance. ISA is a clean, well-motivated architectural contribution: the symmetric 3D-to-2D and 2D-to-3D cross-attention with shared NDC positional encodings is a natural extension of standard attention and is presented in concrete equations with no apparent circularity. The paper also ships practical assets: a video VAE designed for fast human motion, a large mixed real/synthetic training corpus, and a stated commitment to release code and models. If the empirical claims survive matched evaluation, this would be a strong state-of-the-art result for pose-and-camera conditioned human video generation. However, the headline quantitative claim is not fully established by the current comparison protocol, because the classifier-free guidance settings are not matched across methods and the test sets are self-curated and unreleased.

major comments (3)
  1. [Sec. 6.2, Table 2] The statement "we employ CFG=2 while comparison methods are evaluated using their default CFG settings" is an uncontrolled confound. The paper itself observes that CFG=1 produces blurry outputs and that high CFG values produce texture artifacts, so CFG directly affects sharpness, detail, and the reported PSNR/SSIM/LPIPS/FVD values. Under this protocol, the Table 2 margins (e.g., FVD 143.6 vs. 388.2 on the Video set) cannot be attributed to the ISA mechanism. Please rerun the baselines at CFG=2 as well as at a small sweep around it, or tune each method on a validation split and report the chosen values; the SOTA claim is only supported if the advantage persists under matched settings.
  2. [Sec. 6.1, Table 2] The Video, Camera, and Mask test sets are self-curated, comprise 100 videos each, and are not released. No error bars, confidence intervals, or significance tests are reported for any metric. FVD computed on only 100 videos is highly variable, and the absence of per-video or bootstrapped statistics makes it unclear whether the large apparent improvements are robust or driven by dataset idiosyncrasies. Please release the test sets and evaluation code, and report paired per-video standard errors or bootstrap confidence intervals, or supplement with an existing public benchmark under matched settings.
  3. [Sec. 6.2] Because the official AnimateAnyone implementation is not publicly available, the paper compares against the third-party Moore-AnimateAnyone reimplementation. Results from a reimplementation may not reflect the officially published model, which weakens the Table 2 comparison for that baseline in particular. Please report the exact version and checkpoint used, provide the generated outputs, and, where possible, compare with official implementations or with an additional independent reimplementation to confirm that the reported numbers are not implementation-specific.
minor comments (5)
  1. [Abstract and Sec. 7] The abstract and introduction claim "precise control of the camera and body poses," but Sec. 7 documents significant artifacts from SMPL occlusion-estimation errors and from extreme camera movement; please temper the wording or add an explicit qualification.
  2. [Abstract] The phrase "video variation autoencoder" appears in the abstract; the standard term elsewhere in the paper is "variational autoencoder." Please unify the terminology.
  3. [Eq. (13)] Equation (13) contains an extra closing parenthesis in the value argument: the expression V(z_j + PE(s_ndc)) has an unbalanced parenthesis. Please correct the typesetting.
  4. [Sec. 4.2] The loss function is displayed as two numbered equations, (1) and (2), although it is a single objective; please renumber it as one equation for clarity.
  5. [Sec. 6.3, Table 3] The VBench comparison is underspecified: VBench is a text-to-video benchmark, but the paper only says the first frame of the Video Dataset is used as the input image and does not describe how text prompts and evaluation settings were selected for all methods. Please clarify the protocol, including prompt generation and any text dropout settings.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation reduces to its inputs; the method is built from explicit attention equations and external metrics. Minor self-citation in baselines/datasets exists but is not load-bearing; the CFG mismatch is an evaluation confound, not circularity.

full rationale

Walking the derivation chain: the video VAE is trained with L1, perceptual, KL, GAN, and image-decoding regularization losses (Eq. 1) on Kinetics-600 and Human4DiT data; reconstruction quality is measured against Mochi, Cosmos, and CogVideoX on a separately curated 200-video set. The ISA mechanism is defined by explicit equations (Eqs. 8-14) projecting SMPL points and latent pixels into NDC space with sinusoidal positional encodings; it is standard cross-attention with geometric conditioning, not a fitted parameter renamed as a prediction. The ablation (Table 4, Fig. 14) compares ISA variants and 2D ControlNet on validation loss and held-out metrics. The SOTA claim in Table 2 is an empirical comparison, and the paper transparently states that CFG=2 was used for the proposed model while baselines use their default CFG (Sec. 6.2), which is a legitimate experimental-protocol concern but not a circular reduction: no test metric is an algebraic consequence of a fitted input. Human4DiT and CameraCtrl are cited from the same authors, but as baselines and datasets rather than as justifications of ISA's design, so the self-citations are not load-bearing. I find no step where an output is equivalent to an input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The method is empirical; the ledger records the main hand-set hyperparameters, domain assumptions about SMPL and latent representations, and the two algorithmic constructs introduced. No free parameters appear in a mathematical derivation because the paper makes no such derivation.

free parameters (4)
  • CFG scale for ISA-DiT inference = 2
    Chosen for all reported ISA-DiT results; baselines run at their default CFG, so this asymmetry is part of the quantitative comparison (Sec. 6.2).
  • Video VAE compression factors = f_s=8, f_t=4, c=16
    Fixed architecture choices for latent compression; they trade reconstruction fidelity against diffusion training cost and affect all downstream results (Sec. 4).
  • VAE loss weights = not given in main text
    Weights for L1, LPIPS, KL, regularization, and GAN losses are set by hand and reported only in the supplement; they determine latent quality.
  • Inference and sliding-window settings = 30 DDIM steps, window 24, overlap 8
    Used for ISA-DiT and all baselines; affects comparability of metrics but is a protocol choice.
assumptions (4)
  • domain assumption SMPL templates provide sufficiently accurate cross-frame correspondences for the attention mechanism.
    ISA keys and queries are built from SMPL surface points; if the SMPL estimates are wrong, correspondences are wrong. The authors acknowledge this in Sec. 7 and Fig. 17 for inter-character occlusion.
  • domain assumption Projecting 3D SMPL points and 2D latents into shared NDC coordinates yields useful geometric cues beyond learned correspondences.
    This is the core design hypothesis behind ISPE, validated only through the paper's own ablations (Fig. 14, Table 4).
  • domain assumption The curated training corpus (1M real plus 100K synthetic videos) is representative enough for human video generation and is not contaminated by the evaluation sets.
    The paper states the sources but does not quantify overlap between training and the 100-video test sets.
  • domain assumption The video VAE latent space preserves enough identity and motion information for a flow-based DiT to learn from it.
    The VAE is evaluated separately, and latent regularization is shown to reduce diffusion loss, but this is an internal consistency check, not an external guarantee.
invented entities (2)
  • Interspatial Attention (ISA) block
    purpose: Symmetric cross-attention between 3D SMPL tokens and 2D video tokens, replacing 2D SMPL conditioning maps.
    Validated in this paper's ablations and main results; no independent third-party replication or external benchmark has yet demonstrated the mechanism on its own.
  • Interspatial Positional Encoding (ISPE)
    purpose: Relative positional encoding in NDC space that guides where attention looks between 3D and 2D tokens.
    Introduced and tested here; the improvement over ISA-without-ISPE is shown in Fig. 14 and Table 4 only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interspatial Attention for Efficient 4D Human Video Generation." pith.science (2026). https://pith.science/paper/HYG73EFJ

@misc{pith2026250515800,
  author       = {Pith},
  title        = {Pith review of: Interspatial Attention for Efficient 4D Human Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HYG73EFJ}},
  note         = {Machine review of arXiv:2505.15800}
}
read the original abstract

Generating photorealistic videos of digital humans in a controllable manner is crucial for a plethora of applications. Existing approaches either build on methods that employ template-based 3D representations or emerging video generation models but suffer from poor quality or limited consistency and identity preservation when generating individual or multiple digital humans. In this paper, we introduce a new interspatial attention (ISA) mechanism as a scalable building block for modern diffusion transformer (DiT)--based video generation models. ISA is a new type of cross attention that uses relative positional encodings tailored for the generation of human videos. Leveraging a custom-developed video variation autoencoder, we train a latent ISA-based diffusion model on a large corpus of video data. Our model achieves state-of-the-art performance for 4D human video synthesis, demonstrating remarkable motion consistency and identity preservation while providing precise control of the camera and body poses. Our code and model are publicly released at https://dsaurus.github.io/isa4d/.

Figures

Figures reproduced from arXiv: 2505.15800 by the authors.

Figure 1
Figure 1. We introduce interspatial attention as a building block for diffusion transformer–based generative AI models, enabling high-quality video generation of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Last-frame bias. The latent tends to compress the final key frame in each temporal window (center right). After adding the image￾decoding regularization, the latent maintains balanced temporal information distribution across frames (right). 4.1 Architecture We extend the pretrained image VAE from Stable Diffusion 3 (SD3) [Esser et al. 2024] into a 3D architecture to model temporal dynamics in videos. The original SD… view at source ↗
Figure 5
Figure 5. Comparison of latent distribution from different approaches. We visualize the latent distributions on the evaluation videos. Our method yields well-structured latent representations compared to baseline methods. Ours w/o Reg Ours Steps Loss [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (11 more)
Figure 6
Figure 6. Figure 6: Ablation study of latent regularization. We compare training loss curves of diffusion transformers using latents from VAEs trained with and without regularization. human motion and self-occlusion. Our model more effectively pre￾serves structural and high-frequency deta…
Figure 7
Figure 7. Figure 7: Symmetric Interspatial Attention Block. The attention block is a symmetric operation on 3D SMPL tokens and 2D video tokens. Concatenation is indicated by Ç and element-wise addition by +. For 2D video tokens, we project their coordinates in (latent) pixel space s = [𝑠𝑥…
Figure 8
Figure 8. Figure 8: ISA-DiT pipeline. Overview of our diffusion transformer architecture for 4D human generation taking the reference image, SMPL condition, camera poses, and background videos as input. Our framework starts by tokenizing 3D SMPL conditions. In parallel, 2D video tokens (i…
Figure 9
Figure 9. Figure 9: Qualitative comparisons of generated videos. We compare our approach with the best-performing baselines; each of these methods is conditioned on the reference image shown on the left. Our method achieves superior visual quality, particularly in capturing facial express…
Figure 10
Figure 10. Figure 10: Generating videos with controllable camera trajectories. Our model can generate high-quality human videos conditioned on specific camera trajectories (top left insets), effectively transforming video generation into a dynamic view-synthesis system for multi-view human…
Figure 11
Figure 11. Figure 11: Generating multiple characters. Our method synthesizes multi-character videos featuring realistic interactions, such as dancing and boxing [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Human video generation with controlled backgrounds. Our method generates videos by compositing synthesized digital humans with background scenes, achieving consistent lighting and shadow effects based on background conditions. flexibly maps between generated video con…
Figure 13
Figure 13. Figure 13: Additional generated videos. Our ISA-DiT framework generates high-quality videos across diverse domains, spanning upper-body portraits, full-body movements, anime character and multi-characters animations [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Ablation of interspatial attention. We compare validation loss curves for the same DiT architecture using three different conditioning mechanisms: a baseline that only uses the 2D SMPL normal maps for conditioning, ISA without interspatial positional encoding, and int…
Figure 16
Figure 16. Figure 16: ISA with 3D FLAME for expression generation. ISA could be effectively integrated with more precise 3D face models like FLAME to achieve vivid facial generation. Tab. 4 demonstrate qualitative and quantitative results. For videos involving fast movements and complex 3D…
Figure 17
Figure 17. Figure 17: Failure cases. Our method generates photorealistic multi-character videos given accurate SMPL estimations (top left), but fails when inter￾character occlusions are incorrectly estimated (bottom left). Rapid camera movements also introduce background distortions, as sh…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SeqTex: Generate Mesh Textures in Video Sequence

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SeqTex adapts a pretrained video diffusion model to directly generate complete UV texture maps by jointly predicting four multi-view images and the UV map as a five-frame sequence.

Reference graph

Works this paper leans on

41 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [6]

    arXiv:2404.02101 [cs.CV] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo

    CameraCtrl: Enabling Camera Control for Text-to-Video Generation. arXiv:2404.02101 [cs.CV] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. 2023a. Animate any- one: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117 (2023). Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng...

    arXiv 2023

  2. [7]

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher- Shlizerman

    Miradata: A large-scale video dataset with long durations and structured captions.Advances in Neural Information Processing Systems 37 (2024), 48955–48970. Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher- Shlizerman

    work page 2024

  3. [10]

    DreamHuman: Animatable 3D Avatars from Text. (2023). Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Junkun Yuan, Kai W...

    work page 2023

  4. [11]

    https://arxiv.org/abs/2412.03603 Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton

    HunyuanVideo: A Systematic Framework For Large Video Generative Models.arXiv preprint arXiv:2412.03603 (2024). https://arxiv.org/abs/2412.03603 Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton

    arXiv 2024

  5. [13]

    SIGGRAPH Asia) 36, 6 (2017), 194:1–194:17

    Learning a model of facial shape and expression from 4D scans.ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (2017), 194:1–194:17. https://doi.org/10.1145/3130800. 3130813 Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu

    doi:10.1145/3130800 2017

  6. [14]

    ArXiv (Aug 2023)

    TADA! Text to Animatable Digital Avatars. ArXiv (Aug 2023). Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt

    work page 2023

  7. [16]

    arXiv preprint arXiv:2311.17061 (2023)

    HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting. arXiv preprint arXiv:2311.17061 (2023). Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black

    arXiv 2023

  8. [17]

    https://lumalabs.ai/dream-machine

    Luma Dream Machine. https://lumalabs.ai/dream-machine. Accessed: 2025-01-22. Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan

    work page 2025

Show all 41 references
  1. [18]

    arXiv preprint arXiv:2409.04410 (2024)

    Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.04410 (2024). Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black

    2024 arXiv

  2. [19]

    arXiv preprint arXiv:1411.1784 (2014)

    Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014). Mochi-Team

    2014 arXiv

  3. [20]

    arXiv preprint arXiv:2407.02371 (2024)

    Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371 (2024). OpenAI

    2024 arXiv

  4. [21]

    https://openai.com/index/ video-generation-models-as-world-simulators/

    Video generation models as world simulators. https://openai.com/index/ video-generation-models-as-world-simulators/. Accessed: 2024-05-19. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove

    2024

  5. [22]

    arXiv preprint arXiv:2408.00714 (2024)

    Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024). Fitsum Reda, Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, and Ming-Yu Liu

    2024 arXiv

  6. [23]

    https://runwayml.com/

    Runway Gen-3. https://runwayml.com/. Accessed: 2025-01-22. Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu

    2025

  7. [24]

    arXiv preprint arXiv:2405.17405 (2024)

    Human4DiT: Free-view Human Video Generation with 4D Diffusion Transformer. arXiv preprint arXiv:2405.17405 (2024). Aliaksandr Siarohin, Stéphane Lathuilière, Enver Sangineto, and Nicu Sebe. 2019a. Appearance and pose-conditioned human image generation using deformable gans. IE...

    2024 arXiv

  8. [27]

    arXiv preprint arXiv:2410.10306 (2024)

    Animate-x: Universal character image animation with enhanced motion representation. arXiv preprint arXiv:2410.10306 (2024). Zachary Teed, Lahav Lipson, and Jia Deng

    2024 arXiv

  9. [28]

    Advances in Neural Information Processing Systems (2023)

    Deep Patch Visual Odometry. Advances in Neural Information Processing Systems (2023). Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov

    2023

  10. [29]

    arXiv preprint arXiv:2104.15069 (2021)

    A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104.15069 (2021). Zhengyan Tong, Chao Li, Zhaokang Chen, Bin Wu, and Wenjiang Zhou

    2021 arXiv

  11. [30]

    arxiv (2024)

    MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation. arxiv (2024). Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly

    2024

  12. [32]

    arXiv:2408.14837 [cs.LG] https://arxiv.org/ abs/2408.14837 Aaron Van Den Oord, Oriol Vinyals, et al

    Diffusion Models Are Real-Time Game Engines. arXiv:2408.14837 [cs.LG] https://arxiv.org/ abs/2408.14837 Aaron Van Den Oord, Oriol Vinyals, et al

    arXiv

  13. [33]

    Advances in neural information processing systems 30 (2017)

    Neural discrete representation learning. Advances in neural information processing systems 30 (2017). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

    2017

  14. [34]

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan

    Attention is all you need.Advances in neural information processing systems 30 (2017). Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan

    2017

  15. [35]

    arXiv preprint arXiv:2503.20314 (2025)

    Wan: Open and Advanced Large- Scale Video Generative Models. arXiv preprint arXiv:2503.20314 (2025). Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al

    2025 arXiv

  16. [36]

    arXiv preprint arXiv:2410.08260 (2024)

    Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. arXiv preprint arXiv:2410.08260 (2024). Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang

    2024 arXiv

  17. [37]

    arXiv e-prints (2023), arXiv–2307

    Disco: Disentangled control for referring human dance generation in real world. arXiv e-prints (2023), arXiv–2307. Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu

    2023

  18. [38]

    arXiv preprint arXiv:2104.10157 (2021)

    Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 (2021). Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. 2024a. Direct-a-Video: Customized Video Generation with User-Directed Cam...

    2021 arXiv

  19. [39]

    arXiv preprint arXiv:2110.04627 (2021)

    Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 (2021). Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al . 2023a. Magvit: Masked generative video tr...

    2021 arXiv

  20. [40]

    https://arxiv.org/abs/2405.20279 (2024)

    CV-VAE: A Compatible Video VAE for Latent Generative Video Models. https://arxiv.org/abs/2405.20279 (2024). Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. 2023a. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Pro...

    2024 arXiv

  21. [41]

    https://github.com/hpcaitech/Open-Sora Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu

    Open-Sora: Democratizing Efficient Video Production for All. https://github.com/hpcaitech/Open-Sora Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. 2023b. Avatarrex: Real-time expressive full-body avatars. ACM Transactions on Graphics (TOG) 42, 4 (2023),...

    2023

  22. [1171]

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019b. First order motion model for image animation. Advances in neural information processing systems 32 (2019). Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe

    2019

  23. [2013]

    arXiv preprint arXiv:1312.6114 (2013)

    Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013). Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Fieraru, and Cristian Sminchisescu

    2013 arXiv

  24. [2014]

    Advances in neural information processing systems 27 (2014)

    Generative adversarial nets. Advances in neural information processing systems 27 (2014). Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama

    2014

  25. [2016]

    ArXiv e-prints (2016), arXiv–1607

    Layer normalization. ArXiv e-prints (2016), arXiv–1607. Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero

    2016

  26. [2017]

    arXiv preprint arXiv:1705.06950 (2017)

    The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017). Diederik P Kingma

    2017 arXiv

  27. [2019]

    FVD: A new metric for video generation. (2019). Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter

    2019

  28. [2020]

    arXiv:2010.02502 (October 2020)

    Denoising Diffusion Implicit Models. arXiv:2010.02502 (October 2020). https://arxiv.org/abs/2010.02502 Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, and Ming Yang

    2010 arXiv

  29. [2021]

    ACM transactions on graphics (TOG) 40, 6 (2021), 1–16

    Neural actor: Neural free-view synthesis of human actors with pose control. ACM transactions on graphics (TOG) 40, 6 (2021), 1–16. Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu

    2021

  30. [2022]

    Advances in Neural Information Processing Systems 35 (2022), 19900–19916

    Generative neural articulated radiance fields. Advances in Neural Information Processing Systems 35 (2022), 19900–19916. Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan

    2022

  31. [2023]

    arXiv preprint arXiv:2312.06662 (2023)

    Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 (2023). Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang

    2023 arXiv

  32. [2024]

    In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

    2024

  33. [2025]

    arXiv preprint arXiv:2501.03575 (2025)

    Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 (2025). Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.