Pith. sign in

REVIEW 4 major objections 6 minor 7 references

HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single feed-forward transformer, conditioned on rasterized SMPL-X neural textures, reconstructs a person under novel viewpoints and animates them under novel poses from monocular or sparse photos.

desk verdict Solid feed-forward human reconstruction+animation: LVSM + shared SMPL-X neural texture is a genuine new combination; evaluation gaps on identity split and pose robustness temper the claims. read the letter →

arxiv 2506.03118 v1 pith:HLLF547E submitted 2025-06-03 cs.GR cs.CV

Zhiyuan Yu , Zhe Li , Hujun Bao , Can Yang , Xiaowei Zhou This is my paper
classification cs.GRcs.CV
keywords humanreconstructionanimationlargemodelSMPL-Xneuraltexturepose-conditionedrenderingnovelviewsynthesisfeed-forwardtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that human reconstruction and animation do not need dense capture or per-subject optimization: a single feed-forward transformer, conditioned on explicit SMPL-X pose images, can render a person from sparse or monocular photos under both new viewpoints and new poses. HumanRAM computes these pose images by rasterizing a shared learnable neural texture bound to the SMPL-X mesh onto every input and target view, then concatenates them with RGB and camera-ray tokens before the transformer's self-attention. If correct, the same network generalizes to unseen people and unseen poses in one forward pass, which would make drivable avatars practical from casual photos. The paper reports consistent gains over generalizable reconstruction baselines and over generalizable animation baselines on THuman2.1, Human4DiT, ZJUMoCap, and ActorsHQ.

What carries the argument

The load-bearing object is the SMPL-X neural texture: a learnable tri-plane feature field defined in the canonical SMPL-X space and shared across all identities. A registered SMPL-X mesh is rasterized into per-pixel position maps in each input view, and those maps bilinearly sample the tri-planes to produce pose images that are spatially aligned with the RGB frames. The same rasterization is repeated for a target viewpoint and, for animation, for a target pose, so a single texture encodes where the body is in every view and every pose. These pose images, concatenated with RGB and Plücker ray embeddings and patchified, give the transformer explicit correspondences, and a DPT-based decoder fuses neighboring patch tokens to suppress patch artifacts.

What would settle it

Run HumanRAM on an in-the-wild single image using an off-the-shelf SMPL-X fitter and automatic camera calibration, then compare PSNR and LPIPS on held-out real videos against the paper's registered-mesh numbers; a large drop, or visibly broken animation under small pose jitter, would show that the gains come from exact registration rather than from the learned pose-conditioning mechanism.

Watch

Extended reading notes

Core claim

The central claim is that explicit pose conditions, encoded as rasterized SMPL-X neural textures, give a pure-transformer view-synthesis model reconstruction and animation ability at once, without an explicit 3D representation or per-subject fitting. Because the same canonical tri-plane texture is projected through different viewpoints and different SMPL-X poses, the pose images supply spatially aligned correspondences that the transformer's attention can use both to reassemble a novel view and to transfer appearance into a novel pose. On the reported benchmarks the pose-image conditioning raises reconstruction PSNR on THuman2.1 from 28.24 (LVSM) to 30.34 and animation PSNR on ZJUMoCap from 21.29 (NNA) to 23.40, and the model transfers to ActorsHQ without finetuning. The paper presents this as evidence that a single feed-forward model can be both a generalizable reconstructor and an animator.

Load-bearing premise

The method requires an accurate SMPL-X registration and camera calibration for every input image and for every target viewpoint or pose at test time, and the reported numbers all use registered meshes, so the real-world behavior depends on a pose-estimation step the paper does not validate.

Editorial extensions

If this is right

  • One forward pass yields both novel views and novel poses from monocular or sparse input, so neither a dense capture rig nor per-subject optimization is required for the reported scenarios.
  • Adding more input views improves output quality monotonically in the reported numbers (PSNR 21.69 for one view, 32.34 for eight), so the architecture directly benefits from extra observations.
  • Replacing the linear decoder with a DPT-based decoder removes the patch-like artifacts that appear near self-occlusions and thin structures in human renderings.
  • The shared SMPL-X neural texture transfers to real captured data without finetuning, which the paper demonstrates on ActorsHQ where baseline reconstruction methods fail.
  • Animation and reconstruction share one model, so the same weights can drive an unseen person into poses from external motion databases such as AMASS.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If automatic SMPL-X fitting becomes reliable enough, the same architecture could accept pose images from an off-the-shelf fitter and turn single in-the-wild photos into drivable avatars; the paper offers only qualitative in-the-wild results, so this is the natural next test.
  • The pose-image mechanism is not tied to humans: any rasterizable articulated template with a shared neural texture could feed the same transformer, pointing toward animals, hands, or clothed accessories as extensions the paper does not explore.
  • Because the model regresses images rather than a 3D representation, downstream operations like relighting, retexturing, or physical simulation would need an extra stage, so the practical reach is rendering-focused.
  • Jittering or corrupting the SMPL-X registrations and measuring the PSNR drop would quantify how much of the gain comes from accurate geometry priors versus the texture-conditioning itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. HumanRAM proposes a feed-forward transformer model that unifies human reconstruction and pose-controlled animation from monocular or sparse human images. The method builds on LVSM by injecting explicit SMPL-X pose conditions: a learnable shared tri-plane neural texture is rasterized onto input and target views to form pose images, which are concatenated with RGB images and Plücker ray embeddings and fed as tokens to a decoder-only transformer; a DPT-based decoder regresses the target-view and target-pose image. Training uses MSE and perceptual losses. Experiments on THuman2.1, Human4DiT, ZJUMoCap, and ActorsHQ report PSNR, SSIM, and LPIPS against reconstruction baselines (GPS-Gaussian, GHG, LaRa, LVSM) and animation baselines (NNA, SHERF, 3DGS-Avatar), together with ablations of the pose images and the DPT decoder.

Significance. The paper addresses a real gap: it is, to my knowledge, the first feed-forward large reconstruction model that jointly handles sparse-view human reconstruction and novel-pose animation in a single pass. The central idea of rasterizing a shared SMPL-X neural texture into pose-conditioning tokens is clean, well motivated, and supported by the ablations in Table 5, which show that both the pose-image conditioning and the DPT-based decoder contribute to the reported quality. The reported margins over LVSM (e.g., PSNR 30.34 vs 28.24 on THuman2.1 in Table 1) are substantial. The method is not circular: conditioning on a target pose image rendered from the known SMPL-X pose is standard conditional synthesis, and the neural texture is learned end-to-end rather than being a fixed constant. However, the experimental validation is narrower than the abstract claims, because all quantitative evaluations assume registered SMPL-X meshes and calibrated cameras, and the identity-disjointness of the test splits is not established. These issues affect the generality claim but do not undermine the internal consistency of the method.

major comments (4)
  1. [Section 4.2, Table 1, and Fig. 5] The test set is described as 200 randomly selected scans from THuman2.1 and Human4DiT, while the training set uses 2300 THuman2.1 scans; the paper does not state that the test scans are identity-disjoint from the training scans. Consequently, the 'unseen subjects' claims in Fig. 5 and Tables 3-4 are not verifiable from the reported protocol. Please specify the split, ensure there is no identity overlap between training and test scans, or remove the 'unseen subjects' wording.
  2. [Section 3.3, Eqs. (6)-(8), and Section 4.1] The pose images are rasterized from registered SMPL-X meshes, and every quantitative experiment uses dataset-provided SMPL-X registrations and calibrated cameras; in-the-wild results are qualitative only. Thus the abstract's claim of reconstruction and animation 'from monocular or sparse human images' is only validated under privileged test-time information. Please add a quantitative robustness study using an off-the-shelf SMPL-X estimator or perturbed camera/pose inputs on a benchmark such as THuman2.1 or ZJUMoCap, or explicitly restrict the claim to settings with reliable SMPL-X and camera parameters.
  3. [Section 4.3, Tables 3-4] The animation comparisons are asymmetric in their training protocols: NNA is evaluated with official weights without retraining on the same data, 3DGS-Avatar is a per-subject optimization method evaluated here from a single frame, while SHERF and HumanRAM are trained on THuman2.1 and ZJUMoCap. Please report the training data and protocol for each baseline and justify the single-frame 3DGS-Avatar setting; otherwise the reported margins may reflect a training-data or protocol advantage rather than an architectural one.
  4. [Section 4.2, Tables 1-5] All quantitative metrics are reported as single runs with no error bars or significance tests. Because test subjects are randomly selected and some margins are relatively small (e.g., Pose Image + Linear vs Ours in Table 5 gives PSNR 30.07 vs 30.34), the claim that HumanRAM 'significantly surpasses' baselines would be more convincing with variance estimates across test subsets or multiple training seeds.
minor comments (6)
  1. [Section 4.2] The phrase 'randomly select 200 scans from Thuman2.1 and Human4DiT' is ambiguous: it is not clear whether 200 scans are taken from each dataset or 200 in total; please clarify.
  2. [Section 3.4, Eq. (8)] The notation y_i in Eq. (8) denotes intermediate tokens of the i-th layer, whereas earlier y_j denotes output tokens of the transformer; please clarify the indexing and how the four intermediate feature maps are composed in the DPT-based decoder.
  3. [Section 3.3, Eq. (5)] The tri-plane sampling formula F(v;T) concatenates three bilinearly interpolated features, but the handling of pixels outside the SMPL-X mesh in the rasterized pose images is not specified; please state what feature values are used for background pixels.
  4. [Section 4.1] The datasets section states that each training scan is rendered to 60-view images at 512 resolution; please clarify how the four input views and the target view are sampled from these 60 views during training and evaluation.
  5. [Fig. 2 caption] The caption contains the typo 'spare-view'; it should read 'sparse-view'.
  6. [Section 5, Limitation] The limitation paragraph only mentions the quadratic growth of token count with resolution; I suggest adding a sentence acknowledging the dependence on SMPL-X registration and camera calibration, which is a more immediate practical limitation of the current evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pose-conditioned synthesis with external baselines; SMPL-X/camera assumptions are robustness concerns, not circularity.

full rationale

HumanRAM's derivation chain is self-contained against external benchmarks. The method feeds target SMPL-X pose images as input tokens (Eqs. 6-8); generating an image under a supplied target pose is conditional synthesis, not predicting the conditioning variable. The shared neural texture F(v;T) is learned end-to-end and shared across all identities, so it does not encode the target identity's RGB appearance; the transformer must transfer appearance from the input RGB images, and the target pose image is a geometric condition rather than the answer. The comparison baselines LVSM, GPS-Gaussian, GHG, LaRa, NNA, SHERF, and 3DGS-Avatar are external works, and the datasets THuman2.1, Human4DiT, ZJUMoCap, and ActorsHQ are public. No load-bearing result is justified only by a self-citation; citations to the authors' own group appear only as prior datasets or baselines in related work and evaluation. The main caveat, namely test-time reliance on accurate SMPL-X registrations and camera parameters, is an input assumption rather than a circular step.

Assumptions & free parameters 2 free parameters · 6 assumptions · 1 invented entities

The central claim is empirical, so the model's learned weights are not counted as free parameters in the derivation sense. We list the explicit hand-set constants and the unproved assumptions, including reliance on LVSM, SMPL-X registration, camera calibration, synthetic training renders, and a shared neural texture that must generalize across identities and poses. The SMPL-X neural texture is an introduced learned component with no external falsifiable handle; it is best treated as part of the architecture, not as a new physical entity.

free parameters (2)
  • Perceptual loss weight lambda = 1.0
    Set by hand in Eq. (9); not swept or justified, and it directly shapes trained outputs. Minor, but it is an explicit free constant.
  • Neural texture tri-plane resolution and feature dimension = Not specified in main text
    Defines pose image content and token channel count; central to the method, but values are deferred to the supplement, preventing independent reproduction.
assumptions (6)
  • domain assumption LVSM is a working foundation: decoder-only transformers with Plucker ray embeddings regress target-view images given input images and cameras.
    Adopted from Jin et al. 2024 in Section 3.1; HumanRAM builds on it, so any failure of LVSM transfers to the proposed method.
  • domain assumption SMPL-X registrations and camera parameters are available and accurate for all input and target views.
    Section 3.3 uses registered SMPL-X and calibrated cameras; real in-the-wild use would require off-the-shelf pose estimation with unknown error.
  • ad hoc to paper A single shared tri-plane neural texture on canonical SMPL-X provides sufficient correspondence signal across identities and poses.
    Core design premise in Section 3.3; no analysis shows the texture stays identity-neutral or that correspondences are reliable under loose clothing or occlusions.
  • domain assumption THuman2.1 evaluation scans are independent of training identities.
    Required for unseen-subject claims; Section 4.2 randomly selects test scans but does not explicitly state an identity-disjoint split, creating potential leakage.
  • domain assumption Training renders on THuman2.1 (Cycles, 60 views, altitude -45 to 45 degrees, radius 2.0 to 3.0) transfer to real-world data.
    Section 4.1; generalization claims rest on synthetic-to-real transfer, evaluated quantitatively only on ActorsHQ reconstruction and qualitatively in the wild.
  • standard math Differentiable rasterization and bilinear tri-plane sampling behave as specified.
    Used in Eq. (5) and Section 3.3; standard computer graphics operations with well-established implementations.
invented entities (1)
  • Shared SMPL-X neural texture (tri-plane feature map)
    purpose: Rasterized into input and target pose images to condition transformer attention for reconstruction and animation.
    A learned latent with no external falsifiable handle; its efficacy is evidenced only by in-distribution metrics (Tables 1-5), not by any prediction outside the training setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers." pith.science (2026). https://pith.science/paper/HLLF547E

@misc{pith2026250603118,
  author       = {Pith},
  title        = {Pith review of: HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HLLF547E}},
  note         = {Machine review of arXiv:2506.03118}
}
read the original abstract

3D human reconstruction and animation are long-standing topics in computer graphics and vision. However, existing methods typically rely on sophisticated dense-view capture and/or time-consuming per-subject optimization procedures. To address these limitations, we propose HumanRAM, a novel feed-forward approach for generalizable human reconstruction and animation from monocular or sparse human images. Our approach integrates human reconstruction and animation into a unified framework by introducing explicit pose conditions, parameterized by a shared SMPL-X neural texture, into transformer-based large reconstruction models (LRM). Given monocular or sparse input images with associated camera parameters and SMPL-X poses, our model employs scalable transformers and a DPT-based decoder to synthesize realistic human renderings under novel viewpoints and novel poses. By leveraging the explicit pose conditions, our model simultaneously enables high-quality human reconstruction and high-fidelity pose-controlled animation. Experiments show that HumanRAM significantly surpasses previous methods in terms of reconstruction accuracy, animation fidelity, and generalization performance on real-world datasets. Video results are available at https://zju3dv.github.io/humanram/.

Figures

Figures reproduced from arXiv: 2506.03118 by the authors.

Figure 1
Figure 1. We propose HumanRAM, a novel approach for feed-forward novel view synthesis (reconstruction) and novel pose synthesis (animation) from sparse/single-view human image(s). The animation poses are from ActorsHQ [Işık et al. 2023] and AMASS [Mahmood et al. 2019]. ∗The first two authors contributed equally to this work. †Work done during an internship at Huawei. ‡Corresponding authors. Authors’ Contact Information: Zhiyu… view at source ↗
Figure 2
Figure 2. Pipeline of HumanRAM. HumanRAM adopts transformers for human reconstruction and animation from sparse view images in a feed-forward manner. We first patchify and project spare-view RGB images and their corresponding Plücker rays and pose images into input tokens through a linear layer. The pose images are acquired by rasterizing the SMPL-X neural texture onto the input views. Similarly, given the target novel view u… view at source ↗
Figure 3
Figure 3. Illustration of the process of neural texture rasterization. We first render position maps with canonical SMPL-X as vertex colors, and then the position maps are used to sample triplane-based neural texture. Input Tokens. Given calibrated multi-view input images of a char￾acter, we obtain the registered SMPL-X mesh using multi-view mo￾tion capture like [Zhang et al. 2021]. The registered SMPL-X serves as a geometry … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results on ActorsHQ [Işık et al. 2023] and THuman2.1 [Yu et al. 2021]. The top two rows show the reconstruction and animation results from multi-view inputs, while the bottom two rows show the results from single-view input. The driving poses for animation …
Figure 5
Figure 5. Figure 5: Qualitative comparisons for reconstruction on THuman2.1 [Yu et al. 2021] and Human4DiT [Shao et al. 2024a]. We input 4 multi-view images of unseen subjects, and our method achieves a more faithful rendering compared to other reconstruction methods. The first four rows …
Figure 6
Figure 6. Figure 6: Qualitative comparisons for reconstruction on ActorsHQ [Işık et al. 2023]. We input 5 multi-view images, and our method achieves a more faithful rendering compared to other state-of-the-art generalizable reconstruction methods. The experiments are evaluated on the THum…
Figure 7
Figure 7. Figure 7: Qualitative comparisons for multi-view animation on ZJUMo￾Cap [Peng et al. 2021b]. We input 4 multi-view images of the unseen subject, and our method achieves a more photo-realistic rendering com￾pared to other methods. in DPT are helpful in integrating information fro…
Figure 8
Figure 8. Figure 8: Qualitative comparisons for ablations on proposed core com￾ponents. Compared with positions, pose image helps capture detailed structures (indicated by red and blue boxes). Furthermore, the DPT-based decoder helps reduce the patch-like artifacts in regions with severe …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages

  1. [4]

    InEuropean conference on computer vision

    KeypointNeRF: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. InEuropean conference on computer vision. Springer, 179–197. B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. 2020. Nerf: Representing scenes as neural radiance fields for view synthesis. InECCV. Gyeongsik Moon, Takaaki Shirato...

    work page 2023

  2. [2008]

    DiHuR: Diffusion-Guided Generalizable Human Reconstruction

    Markerless garment capture. InACM SIGGRAPH. 1–9. Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong. 2024. Dreama- vatar: Text-and-shape guided 3d human avatar generation via diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 958–968. Yukang Cao, Kai Han, and Kwan-Yee K. Wong. 2023. SeSDF:...

    work page Pith review arXiv 2024

  3. [2011]

    In2011 International Conference on Computer Vision

    Shading-based dynamic shape refinement from multi-view video under general illumination. In2011 International Conference on Computer Vision. IEEE, 1108–1115. Junjin Xiao, Qing Zhang, Zhan Xu, and Wei-Shi Zheng. 2024. NECA: Neural Customiz- able Human Avatar. InCVPR. Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, X...

    arXiv 2024

  4. [2020]

    Convolutional occupancy networks. InECCV. Springer, 523–540. Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2024. Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos.TPAMI(2024). Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xia...

    arXiv 2024

  5. [2021]

    Motion Representations for Articulated Animation. InCVPR. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556(2014). Jonathan Starck and Adrian Hilton. 2007. Surface capture for performance-based animation.IEEE computer graphics and applications27, 3 (2007), 21–31. Sh...

    arXiv 2014

  6. [2022]

    IEEE Transactions on Pattern Analysis and Machine Intelligence(2022), 1–12

    MPS-NeRF: Generalizable 3D Human Rendering From Multiview Images. IEEE Transactions on Pattern Analysis and Machine Intelligence(2022), 1–12. https: //doi.org/10.1109/TPAMI.2022.3205910 Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, et al. 2019. The religh...

    work page arXiv 2022

  7. [2023]

    Naureen Mahmood, Nima Ghorbani, Nikolaus F

    Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos.arXiv preprint arXiv:2304.01186(2023). Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019. AMASS: Archive of Motion Capture as Surface Shapes. InInternational Conference on Computer Vision. 5442–5451. Yifang Men, Yuan Yao, Miaomiao Cu...

    arXiv 2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.