Pith. sign in

REVIEW 4 major objections 5 minor 86 references

A video diffusion model can act as a neural shader for portrait animation when conditioned on dense, pixel-aligned 3D face normal maps rather than sparse landmarks or implicit motion latents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 22:57 UTC pith:D4EP55EQ

load-bearing objection Solid, well-ablated systems paper; the dense 3DMM-normal-map conditioning is a real integration, but the motion metrics don't back the abstract's expression-control claim. the 4 major comments →

arxiv 2607.24124 v1 pith:D4EP55EQ submitted 2026-07-27 cs.CV

ViDS: Video Diffusion Shader using 3D Face Tracking

Wenbo Ji , Davide Davoli , Zhe Chen , Liam Schoneveld , Matthias Nie{\ss}ner , Jiapeng Tang This is my paper
classification cs.CV
keywords portrait animationvideo diffusion3D morphable modelnormal map conditioningidentity preservationface reenactmentautoregressive video generationclassifier-free guidance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ViDS sets out to prove that a video diffusion model can act as a neural shader for portrait animation when it is given dense geometric guidance from a 3D Morphable Model tracker. The core proposal is to render pixel-aligned normal maps from an identity-frozen 3D face mesh animated by a driving video, and concatenate them with the reference image and noisy latents into a pretrained video diffusion transformer. The authors argue this yields finer and more consistent expression and pose control than landmark-based or implicit-latent diffusion baselines, while preserving reference identity and appearance — including non-3DMM regions such as inner mouth and hair. The claim is supported by self- and cross-reenactment benchmarks on VFHQ and Celeb-V-Text, a user study, and ablations that isolate the conditioning signal and tracking quality.

Core claim

ViDS claims that accurate monocular 3DMM tracking — specifically the Pixel3DMM tracker — converts expression and pose transfer into a shading problem. An identity-specific mesh is reconstructed from one reference image, then animated with a driving video's pose and expression parameters while shape parameters are frozen to prevent identity leakage; the animated mesh is rendered as normal maps and fed, together with the reference image and a text prompt, into a video diffusion transformer. The study reports that this dense geometric conditioning outperforms sparse-landmark and implicit-latent methods on most metrics in self- and cross-reenactment, with a particular advantage in identity prese

What carries the argument

The central object is the rendered pixel-aligned normal-map sequence from an animated 3DMM mesh. Unlike low-dimensional 3DMM parameters or sparse landmarks, these maps carry dense local surface orientation that is lighting-agnostic and frame-aligned with the reference image; freezing the reference identity shape separates identity from pose and expression. This geometry channel is injected into a pretrained video diffusion transformer via a unified channel-concatenation layer (reference latents, normal-map latents, noisy video latents), and three separate classifier-free guidance branches (identity, geometry, text) are combined into one velocity field. Long sequences are produced by an autor

Load-bearing premise

The whole pipeline leans on the assumption that Pixel3DMM tracking is accurate enough that its rendered normal maps faithfully capture the driving video's pose and expression — the paper's own conclusion lists tracking sensitivity as a limitation, and the ablation shows that swapping in SHeaP tracking sharply degrades identity and geometry metrics.

What would settle it

Run the VFHQ self- and cross-reenactment evaluation while progressively corrupting or replacing the 3DMM tracking (e.g., jittering pose parameters or substituting SHeaP), and check whether identity and geometry metrics degrade as sharply as Table 8 suggests; if a corrupted tracking signal still yields strong CSIM and AKD, the conditioning is not doing the claimed work. A second decisive check is to re-evaluate all methods with a non-3DMM motion-fidelity metric or independent human ratings on expression transfer, since AED/APD share the paper's geometric prior.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dense geometric conditioning becomes a workable alternative to landmarks or implicit latents for one-shot portrait animation.
  • Improvements in monocular 3D face tracking quality should translate directly into finer expression and pose transfer, since the normal maps are the sole motion channel.
  • Identity leakage from the driving video is reduced by freezing the reference identity's 3DMM shape parameters, which should help cross-identity and in-the-wild reenactment.
  • The autoregressive overlapping-window scheme extends a pretrained video diffusion model beyond its native temporal window while reducing boundary discontinuities.
  • The method retains photorealistic synthesis of regions the 3DMM does not model, such as inner mouth and hair, because the diffusion prior still generates those details.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation list (tracking sensitivity, slow autoregressive inference, no relighting, limited control beyond 3DMM regions) points to the clearest next steps: more robust tracking and faster sampling would expand the method's practical range.
  • Because the reported motion metrics (AED/APD) are computed with a 3DMM estimator, a re-ranking of methods using a different estimator or purely perceptual motion judgments would test whether the advantage is genuine or partly an artifact of shared geometric priors.
  • A testable extension: swapping the 3DMM tracker at inference for a stronger one should improve the same metrics on the same benchmarks, providing a direct way to measure how much headroom remains in tracking quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ViDS, a portrait animation method that uses Pixel3DMM/FLAME normal maps as dense geometric conditioning for a Wan-based video diffusion model. Identity shape is fixed from the reference image, driving pose/expression animates the mesh, and normal maps are concatenated with reference latents through a unified conditioning layer. Multi-CFG with separate identity/geometry/text guidance and autoregressive overlapping-window inference are used. Experiments on VFHQ and Celeb-V-Text compare to Follow-Your-Emoji, X-Portrait, HunyuanPortrait, and Wan-Animate, with ablations on conditioning signals, architecture, guidance scale, training size, and tracker. The paper reports best or second-best results on most reconstruction and identity metrics, and a user study favors ViDS on all dimensions.

Significance. If the evidence held, ViDS would be a useful and simple design: it injects dense 3DMM geometry into a pretrained video diffusion model with minimal architectural change, and it demonstrates the importance of tracking quality. The ablations are extensive and the training protocol is described in enough detail to reproduce. The main weakness is that the advertised advantage in expression/pose control is not consistently supported by the numerical motion metrics, and the motion metrics themselves share a 3DMM prior with the conditioning signal. The central claim is thus plausible but not conclusively demonstrated.

major comments (4)
  1. [Sec. 4.1, Tables 1 and 2] The abstract claims 'more detailed and consistent expression and pose control', but the only direct expression metric (AED) never favors ViDS. In Table 1 self-reenactment AED is 0.121 for Ours versus 0.113 (HunyuanPortrait), 0.118 (X-Portrait), and 0.118 (Wan-Animate); cross-reenactment AED is 0.298 versus 0.279 (HunyuanPortrait). In Table 2, Ours again has worse AED in both self (0.189 vs 0.161) and cross (0.306 vs 0.298) settings. This should be confronted directly: either temper the claim, show that AED is the wrong yardstick with a validated alternative, or provide a significance test showing the differences are not meaningful.
  2. [Sec. 4, Evaluation Protocol; Sec. 3.1] AED and APD are computed with Deep3DFaceRecon [18], a monocular 3DMM estimator, while the conditioning signal is a rendered 3DMM (Pixel3DMM/FLAME) normal map. This creates a circularity concern: outputs that stay close to the 3DMM manifold may score better on these metrics even if they are not perceptually more faithful. The paper acknowledges the limitation and adds a user study, but the user study is too small (10 videos, 40 participants, no error bars or significance tests) to carry the load. I ask for (a) a non-3DMM motion metric (e.g., landmark-velocity or optical-flow-based expression/pose distance) or a second 3DMM estimator with a different topology, and (b) a per-item or paired analysis showing that automated motion metrics agree with the user-study 'expression/pose consistency' ratings.
  3. [Sec. 4, Tables 1-8] No error bars, confidence intervals, or significance tests are reported anywhere. Several headline margins are very small: Table 1 self PSNR 21.19 vs 21.13, CSIM 0.879 vs 0.876, LPIPS 0.178 vs 0.187; Table 2 self PSNR 19.98 vs 19.01. With 50 test sequences, these differences may lie within noise. Please report per-metric standard deviations and paired significance tests (or bootstrap CIs) for at least the main comparison tables and for the key ablations in Table 4.
  4. [Sec. 4, Dataset and Evaluation Protocol] The cross-reenactment protocol is under-specified: 'pairs the first frame of one test sequence as the reference identity with a different sequence as the driving video' does not state how many pairs are used, whether all ordered pairs are evaluated, or how the random pairing is fixed. Since cross-reenactment numbers depend strongly on the chosen pairs, please specify the exact evaluation set and report the number of generated videos.
minor comments (5)
  1. [Tables 1, 2, 4, 6, 7] The column 'IQA' is never defined. Please state which image quality assessment is used and how it is computed.
  2. [Table 7] The sentence 'Overall performance improves with scale despite metric-level fluctuations' is not supported by the table: several metrics degrade from 5K to 20K and improve only at 30K, while FID/FVD and IQA fluctuate non-monotonically.
  3. [Table 3] The user-study questionnaire uses a single 'Expression & Pose Consistency' rating. Because the paper's central claim concerns expression and pose separately, please either separate these two questions or justify merging them.
  4. [Sec. 4.2, Table 8] The tracking-quality comparison reports only CSIM/AKD and not AED/APD; adding AED/APD would directly support the sentence 'Accurate tracking is therefore important for identity and geometry control.'
  5. [Sec. 3.3] The autoregressive windowing description says 'nominal overlap of f frames' and 'stride h=s-f', but the pseudo-code/algorithm for tail alignment and pixel-space blending is not given. A short algorithm box or pseudo-code would help reproducibility.

Circularity Check

0 steps flagged

No significant circularity; the derivation is self-contained and benchmarked against external baselines.

full rationale

ViDS's central claim is that dense 3DMM normal maps as conditioning produce more detailed expression/pose control while preserving identity. The derivation chain is a standard supervised fine-tuning setup: Eq. (1) trains a flow-matching velocity field from noisy latents, conditions, and clean latents; Eq. (4) combines classifier-free guidance branches; Eq. (2) defines generation as a function of the source image, normal maps, and prompt. None of these equations defines the predicted video in terms of the evaluation metrics (AED/APD/CSIM), and no evaluation metric is used as a training target. The closest concern is that AED/APD are computed with Deep3DFaceRecon, a 3DMM estimator, while the method conditions on Pixel3DMM/FLAME normal maps. This is an evaluation-confound risk, not a circular reduction: the metrics are computed post hoc on final generated videos by an external estimator, applied identically to all baselines, and the paper acknowledges the limitation ('automated reconstruction can misalign with perceived quality') and supplements with a user study that does not involve 3DMM parameter estimation. The self-citations to Pixel3DMM and SHeaP are used as interchangeable tracking modules with an explicit sensitivity ablation (Table 8), not as authoritative proof of the central claim; no load-bearing argument reduces to an unverified self-citation. The paper is evaluated on external benchmarks (VFHQ, Celeb-V-Text) against external baselines, so the central comparison is not constructed from the method's own assumptions. Accordingly, no circular step meeting the evidence threshold (quote plus specific reduction) is present; the minor self-citation weight is reflected in the score of 1 rather than 0.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper's central claim leans on: (i) the accuracy of a self-developed/third-party 3DMM tracker to produce the normal-map control signal; (ii) FLAME's expressiveness to encode desired motion; (iii) the Wan pretrained prior to render geometry into realistic video; and (iv) the validity of 3DMM-derived evaluation metrics. The reader effectively pays for these assumptions upstream; the paper provides ablations supporting (i) and (iii) but no independent ground-truth 3D evaluation for (i), and the AED/APD metrics are coupled to the conditioning modality.

free parameters (2)
  • CFG guidance scales (w_i, w_g, w_p) = 2.0, 1.5, 6.0
    Selected by ablation in Table 6; used in Eq. (4) to balance identity, geometry, and text conditioning. Different values degrade or over-constrain motion.
  • Class-conditioning dropout probability = 0.10
    Set empirically in Sec. 3.2 to enable multiple classifier-free guidance; directly controls how strongly each conditioning branch is learned.
axioms (4)
  • domain assumption Pixel3DMM [25] provides accurate and stable monocular 3D face reconstruction and expression tracking on the evaluation videos.
    The entire control signal N is rendered from these tracked parameters; Table 8 shows replacing Pixel3DMM with SHeaP degrades results, supporting but not proving accuracy.
  • domain assumption FLAME's parametric shape/expression space is expressive enough to represent the facial motions that matter for portrait animation.
    If FLAME cannot represent some expressions or poses, normal-map conditioning cannot convey them; the paper shows extreme expressions but no coverage analysis.
  • domain assumption The Wan video diffusion prior, after fine-tuning, can synthesize photorealistic faces and follow normal-map conditioning without relying on spurious shortcuts.
    The method assigns motion control to normal maps, but the reference image and text prompt also enter the model; ablations show image/geometry are essential, but cannot prove the model does not also use text or reference cues for motion.
  • domain assumption The evaluation metrics (PSNR/SSIM/LPIPS/CSIM/AED/APD/AKD/FID/FVD/IQA) and the user study are reliable proxies for identity preservation and motion accuracy.
    In particular, AED/APD are computed with Deep3DFaceRecon [18], a 3DMM-based estimator; the paper itself flags misalignment between automated reconstruction and perceived quality.

pith-pipeline@v1.3.0-alltime-deepseek · 16222 in / 14809 out tokens · 126805 ms · 2026-07-31T22:57:01.017580+00:00 · methodology

0 comments
read the original abstract

We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/

Figures

Figures reproduced from arXiv: 2607.24124 by Davide Davoli, Jiapeng Tang, Liam Schoneveld, Matthias Nie{\ss}ner, Wenbo Ji, Zhe Chen.

Figure 1
Figure 1. Figure 1: Given a single face image (first column), our method synthesizes vivid facial animations (second to sixth columns) guided by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline Overview. We introduce a video diffusion shader for controllable portrait animation from a single image. Given a source image and a driving sequence, the goal is to generate a portrait video that preserves the identity and appearance of the source while following the pose and expression changes in the driving video. We achieve this by using 3DMM tracking as the conditioning signal. We first recons… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparisons. From top to bottom, we show self-reenactment, cross-reenactment, and in-the-wild generalization. In all three settings, our method outperforms the baselines in handling both extreme head poses and challenging facial expressions, achieving better identity preservation, higher visual quality, and more accurate expression and pose transfer. Additional Evaluation on Celeb-V-Text. To fu… view at source ↗
Figure 4
Figure 4. Figure 4: Generation beyond rigid 3DMM constraints. (a) Faithful inner-mouth details, such as teeth and tongue. (b) Text￾prompted hair dynamics. in non-parametric regions. As shown in Fig. 4a and Fig. 4b, our method generates realistic inner-mouth details such as teeth and tongue, handles extreme poses and expressions, and uses text prompts to complement 3DMM guidance for complex hair dynamics. User Study. We conduc… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation on conditioning signals. Comparison of dif￾ferent control signals for expression and pose control. etry over a fixed-topology surface, which explains their best overall identity preservation and expression-transfer perfor￾mance in Tab. 4, despite a few metric-specific advantages from other representations. Reference shape parameters. The 3DMM shape param￾eters of the reference image provide critic… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on reference shape parameters. Without identity swapping, using the driving subject’s shape parameters produces inaccurate proportions and contours (left). Replacing them with the reference identity parameters restores the correct facial structure (right). Classifier-free guidance weights. We ablate guidance scales (wi , wg, wp) for identity, geometry, and text. In Tab. 6, lowering wp to 4.0 weake… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 16 linked inside Pith

  1. [1]

    Neural point-based graph- ics

    Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graph- ics. InEuropean conference on computer vision, pages 696–

  2. [2]

    A morphable model for the synthesis of 3d faces

    V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. InProceedings of the 26th Annual Conference on Computer Graphics and Interactive Tech- niques, SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, pages 187–194. ACM, 1999

    1999

  3. [3]

    A groupwise multilinear correspondence optimization for 3d faces

    Timo Bolkart and Stefanie Wuhrer. A groupwise multilinear correspondence optimization for 3d faces. InProceedings of the IEEE international conference on computer vision, pages 3604–3612, 2015

    2015

  4. [4]

    A 3d morphable model learnt from 10,000 faces

    James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5543–5552, 2016

    2016

  5. [5]

    3d face morphable models” in-the- wild”

    James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou. 3d face morphable models” in-the- wild”. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 48–57, 2017

    2017

  6. [6]

    Multi- linear wavelets: A statistical shape space for human faces

    Alan Brunton, Timo Bolkart, and Stefanie Wuhrer. Multi- linear wavelets: A statistical shape space for human faces. InComputer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceed- ings, Part I 13, pages 297–312. Springer, 2014

    2014

  7. [7]

    How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). InProceedings of the IEEE international conference on computer vision, pages 1021–1030, 2017

    2017

  8. [8]

    Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

    Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

    2013

  9. [9]

    Video- dreamer: Customized multi-subject text-to-video generation with disen-mix finetuning, 2023

    Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu. Video- dreamer: Customized multi-subject text-to-video generation with disen-mix finetuning, 2023

    2023

  10. [10]

    Invertible Neural BRDF for Object Inverse Rendering .IEEE Trans- actions on Pattern Analysis & Machine Intelligence, 44(12): 9380–9395, 2022

    Zhe Chen, Shohei Nobuhara, and Ko Nishino. Invertible Neural BRDF for Object Inverse Rendering .IEEE Trans- actions on Pattern Analysis & Machine Intelligence, 44(12): 9380–9395, 2022

    2022

  11. [11]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 2403–2410, 2025

    2025

  12. [12]

    Wan-animate: Unified character animation and replacement with holistic replication.arXiv preprint arXiv:2509.14055, 2025

    Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jinwei Qi, Penchong Qiao, et al. Wan-animate: Unified character animation and replacement with holistic replication.arXiv preprint arXiv:2509.14055, 2025

    arXiv 2025

  13. [13]

    Generalizable and ani- matable gaussian head avatar.Advances in Neural Informa- tion Processing Systems, 37:57642–57670, 2024

    Xuangeng Chu and Tatsuya Harada. Generalizable and ani- matable gaussian head avatar.Advances in Neural Informa- tion Processing Systems, 37:57642–57670, 2024

    2024

  14. [14]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.arXiv preprint arXiv:2304.09151, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Emoca: Emotion driven monocular face capture and animation

    Radek Dan ˇeˇcek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 20311–20322, 2022

    2022

  16. [16]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 4690–4699, 2019

    2019

  17. [17]

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 285–295, 2019

    2019

  18. [18]

    Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition workshops, pages 0–0, 2019

    2019

  19. [19]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alek- sei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

    2022

  20. [20]

    Black, and Timo Bolkart

    Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images.ACM Transactions on Graphics (TOG), 40:1 – 13, 2020

    2020

  21. [21]

    Hu- mandit: Pose-guided diffusion transformer for long-form hu- man motion video generation.ArXiv, abs/2502.04847, 2025

    Qijun Gan, Yi Ren, Chen Zhang, Zhenhui Ye, Pan Xie, Xi- ang Yin, Zehuan Yuan, Bingyue Peng, and Jianke Zhu. Hu- mandit: Pose-guided diffusion transformer for long-form hu- man motion video generation.ArXiv, abs/2502.04847, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Deferred neural lighting: free-viewpoint re- lighting from unstructured photographs.ACM Trans

    Duan Gao, Guojun Chen, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. Deferred neural lighting: free-viewpoint re- lighting from unstructured photographs.ACM Trans. Graph., 39(6), 2020

    2020

  23. [23]

    Learning neural parametric head models

    Simon Giebenhain, Tobias Kirschstein, Markos Georgopou- los, Martin R ¨unz, Lourdes Agapito, and Matthias Nießner. Learning neural parametric head models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21003–21012, 2023

    2023

  24. [24]

    Mononphm: Dynamic head reconstruction from monocular videos

    Simon Giebenhain, Tobias Kirschstein, Markos Georgopou- los, Martin R ¨unz, Lourdes Agapito, and Matthias Nießner. Mononphm: Dynamic head reconstruction from monocular videos. InProc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  25. [25]

    Pixel3dmm: Versatile 9 screen-space priors for single-image 3d face reconstruction, 2025

    Simon Giebenhain, Tobias Kirschstein, Martin R ¨unz, Lour- des Agapito, and Matthias Nießner. Pixel3dmm: Versatile 9 screen-space priors for single-image 3d face reconstruction, 2025

    2025

  26. [26]

    Liveportrait: Efficient portrait animation with stitching and retargeting control.ArXiv, abs/2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Dingyun Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control.ArXiv, abs/2407.03168, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis. In2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5764–

    2021

  28. [28]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Belongie, and Ming- Yu Liu

    Zekun Hao, Arun Mallya, Serge J. Belongie, and Ming- Yu Liu. Gancraft: Unsupervised 3d neural rendering of minecraft worlds.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14052–14062, 2021

    2021

  30. [30]

    Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Lam: Large avatar model for one-shot animatable gaus- sian head

    Yisheng He, Xiaodong Gu, Xiaodan Ye, Chao Xu, Zhengyi Zhao, Yuan Dong, Weihao Yuan, Zilong Dong, and Liefeng Bo. Lam: Large avatar model for one-shot animatable gaus- sian head. InProceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pages 1–13, 2025

    2025

  32. [32]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Pith/arXiv arXiv 2022

  33. [33]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffu- sion models.ArXiv, abs/2204.03458, 2022. [34]A 3D Face Model for Pose and Illumination Invariant Face Recognition, Genova, Italy, 2009. IEEE

    Pith/arXiv arXiv 2022

  34. [35]

    3d gaussian splatting for real-time radiance field rendering.ACM Trans

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023

    2023

  35. [36]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    Pith/arXiv arXiv 2023

  36. [37]

    Nersemble: Multi-view ra- diance field reconstruction of human heads.ACM Trans

    Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersemble: Multi-view ra- diance field reconstruction of human heads.ACM Trans. Graph., 42(4), 2023

    2023

  37. [38]

    Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and ex- pression from 4D scans.ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):194:1–194:17, 2017

    2017

  38. [39]

    Learning a model of facial shape and expression from 4d scans.ACM Trans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans.ACM Trans. Graph., 36(6):194–1, 2017

    2017

  39. [40]

    Diffusion- renderer: Neural inverse and forward rendering with video diffusion models

    Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nan- dita Vijaykumar, Sanja Fidler, and Zian Wang. Diffusion- renderer: Neural inverse and forward rendering with video diffusion models. InThe IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2025

    2025

  40. [41]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  41. [42]

    Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines.arXiv preprint arXiv:1906.08172, 2019

    Pith/arXiv arXiv 1906

  42. [43]

    Follow your pose: Pose- guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 4117–4125, 2024

    2024

  43. [44]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024

    2024

  44. [45]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021

    2021

  45. [46]

    Instant neural graphics primitives with a mul- tiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022

    Thomas M ¨uller, Alex Evans, Christoph Schied, and Alexan- der Keller. Instant neural graphics primitives with a mul- tiresolution hash encoding.ACM transactions on graphics (TOG), 41(4):1–15, 2022

    2022

  46. [47]

    Deepsdf: Learning con- tinuous signed distance functions for shape representation

    Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 165–174, 2019

    2019

  47. [48]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205, 2023

    2023

  48. [49]

    Combining 3d morphable models: A large scale face-and-head model

    Stylianos Ploumpis, Haoyang Wang, Nick Pears, William AP Smith, and Stefanos Zafeiriou. Combining 3d morphable models: A large scale face-and-head model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10934–10943, 2019

    2019

  49. [50]

    Joker: Conditional 3d head syn- thesis with extreme facial expressions, 2024

    Malte Prinzler, Egor Zakharov, Vanessa Sklyarova, Berna Kabadayi, and Justus Thies. Joker: Conditional 3d head syn- thesis with extreme facial expressions, 2024

    2024

  50. [51]

    3d facial expressions through analysis-by- neural-synthesis

    George Retsinas, Panagiotis P Filntisis, Radek Danecek, Victoria F Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 3d facial expressions through analysis-by- neural-synthesis. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 2490–2501, 2024

    2024

  51. [52]

    High-resolution image 10 synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image 10 synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  52. [53]

    Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J. Black. Learning to regress 3d face shape and expression from an image without 3d supervision.2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 7755–7764, 2019

    2019

  53. [54]

    Sheap: Self-supervised head geometry predictor learned via 2d gaussians.ArXiv, abs/2504.12292, 2025

    Liam Schoneveld, Zhe Chen, Davide Davoli, Jiapeng Tang, Saimon Terazawa, Ko Nishino, and Matthias Nießner. Sheap: Self-supervised head geometry predictor learned via 2d gaussians.ArXiv, abs/2504.12292, 2025

    arXiv 2025

  54. [55]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. InConference on Neural Information Pro- cessing Systems (NeurIPS), 2019

    2019

  55. [56]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.CoRR, abs/2211.12368, 2022

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.CoRR, abs/2211.12368, 2022

    Pith/arXiv arXiv 2022

  56. [57]

    Dphms: Diffusion paramet- ric head models for depth-based tracking

    Jiapeng Tang, Angela Dai, Yinyu Nie, Lev Markhasin, Jus- tus Thies, and Matthias Nießner. Dphms: Diffusion paramet- ric head models for depth-based tracking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1111–1122, 2024

    2024

  57. [58]

    Gaf: Gaussian avatar reconstruction from monocular videos via multi-view diffu- sion

    Jiapeng Tang, Davide Davoli, Tobias Kirschstein, Liam Schoneveld, and Matthias Niessner. Gaf: Gaussian avatar reconstruction from monocular videos via multi-view diffu- sion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5546–5558, 2025

    2025

  58. [59]

    3D face tracking from 2D video through iterative dense UV to image flow

    Felix Taubner, Prashant Raina, Mathieu Tuli, Eu Wern Teh, Chul Lee, and Jinmiao Huang. 3D face tracking from 2D video through iterative dense UV to image flow. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1227–1237, 2024

    2024

  59. [60]

    Saragih, Matthias Nießner, Rohit Pandey, S

    Ayush Kumar Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason M. Saragih, Matthias Nießner, Rohit Pandey, S. Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhofer. State of the art on neural renderi...

    2020

  60. [61]

    Real- time expression transfer for facial reenactment.ACM Trans

    Justus Thies, Michael Zollh ¨ofer, Matthias Nießner, Levi Val- gaerts, Marc Stamminger, and Christian Theobalt. Real- time expression transfer for facial reenactment.ACM Trans. Graph., 34(6):183–1, 2015

    2015

  61. [62]

    Face2face: Real-time face capture and reenactment of rgb videos.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, 2016

    Justus Thies, Michael Zollh ¨ofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, 2016

    2016

  62. [63]

    Face2face: Real-time face capture and reenactment of rgb videos

    Justus Thies, Michael Zollhofer, Marc Stamminger, Chris- tian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016

    2016

  63. [64]

    Headon: Real-time reen- actment of human portrait videos.ACM Transactions on Graphics (TOG), 37(4):1–13, 2018

    Justus Thies, Michael Zollh ¨ofer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. Headon: Real-time reen- actment of human portrait videos.ACM Transactions on Graphics (TOG), 37(4):1–13, 2018

    2018

  64. [65]

    De- ferred neural rendering: image synthesis using neural tex- tures.ACM Trans

    Justus Thies, Michael Zollh ¨ofer, and Matthias Nießner. De- ferred neural rendering: image synthesis using neural tex- tures.ACM Trans. Graph., 38(4), 2019

    2019

  65. [66]

    Nonlinear 3d face morphable model

    Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7346–7355, 2018

    2018

  66. [67]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fan...

    Pith/arXiv arXiv 2025

  67. [68]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021

    2021

  68. [69]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  69. [70]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024

    Pith/arXiv arXiv 2024

  70. [71]

    3d face reconstruction with dense landmarks

    Erroll Wood, Tadas Baltru ˇsaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevi´c, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljkovi´c, et al. 3d face reconstruction with dense landmarks. InEuropean Confer- ence on Computer Vision, pages 160–177. Springer, 2022

    2022

  71. [72]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657–666, 2022

    2022

  72. [73]

    X-portrait: Expressive portrait anima- tion with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait anima- tion with hierarchical motion attention. InACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  73. [74]

    Make- your-video: Customized video generation using textual and structural guidance.arXiv preprint arXiv:2306.00943, 2023

    Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Make- your-video: Customized video generation using textual and structural guidance.arXiv preprint arXiv:2306.00943, 2023

    Pith/arXiv arXiv 2023

  74. [75]

    11 Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. 11 Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Pith/arXiv arXiv 2024

  75. [76]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.ArXiv, abs/2404.10667, 2024

    Sicheng Xu, Guojun Chen, Yufeng Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.ArXiv, abs/2404.10667, 2024

    Pith/arXiv arXiv 2024

  76. [77]

    Hunyuanportrait: Implicit condition con- trol for enhanced portrait animation

    Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, et al. Hunyuanportrait: Implicit condition con- trol for enhanced portrait animation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 15909–15919, 2025

    2025

  77. [78]

    i3dmm: Deep implicit 3d morphable model of human heads

    Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans- Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. i3dmm: Deep implicit 3d morphable model of human heads. InCVPR, 2021

    2021

  78. [79]

    Renderformer: Transformer-based neural rendering of triangle meshes with global illumination

    Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong. Renderformer: Transformer-based neural rendering of triangle meshes with global illumination. InACM SIG- GRAPH 2025 Conference Papers, 2025

    2025

  79. [80]

    Personatalk: Bring attention to your persona in vi- sual dubbing

    Longhao Zhang, Shuang Liang, Zhipeng Ge, and Tianshu Hu. Personatalk: Bring attention to your persona in vi- sual dubbing. InSIGGRAPH Asia 2024 Conference Papers, pages 1–9, 2024

    2024

  80. [81]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018

    2018

Showing first 80 references.