Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Joint Learning of Depth and Appearance for Portrait Image Animation

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single diffusion pass can generate a face and its matching depth together.

desk verdict A useful joint RGB-depth portrait diffusion system with a real evaluation gap: the headline depth-accuracy claim is unverifiable without an identity-disjoint split. read the letter →

arxiv 2501.08649 v1 pith:S5PJCPGD submitted 2025-01-15 cs.CV cs.LG

Xinya Ji , Gaspard Zoss , Prashanth Chandran , Lingchen Yang , Xun Cao , Barbara Solenthaler , Derek Bradley This is my paper · ORCID
classification cs.CVcs.LG
keywords portraitimageanimationjointdepthandappearancegenerationlatentdiffusionmodelmonocularfacialestimationaudio-driventalkingheadrelightingdepth-conditionedediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that a portrait generator need not choose between appearance and geometry: a single diffusion model can be trained to produce the RGB face and its depth map as one jointly sampled output, so the two always correspond. It argues this by building a latent diffusion backbone whose denoising network is expanded to process RGB and depth latents simultaneously, conditioned on identity features from a reference photo. If this works, one trained model can serve as a depth estimator, a depth-guided face editor, a relighter, and an audio-driven talking-head generator without sacrificing consistency between image and 3D structure. The paper's quantitative evidence is a studio depth-estimation result where the joint model reaches 0.162 absolute relative error, beating a 1B-parameter human-centric foundation model at 0.197.

What carries the argument

The load-bearing component is the channel-expanded latent diffusion backbone: the input and output layers of a pretrained UNet are widened from three latent channels to six, the extra input weights initialized to zero and the extra output weights duplicated, so the pretrained prior is largely preserved while the network learns to denoise RGB and depth together. A ReferenceNet, initialized from the same backbone, extracts identity features from the reference photo and injects them via spatial attention. For applications, asymmetric inpainting masks on the RGB and depth branches turn the joint model into a bidirectional conditional generator, and audio-attention and temporal-attention modules extend it to video.

What would settle it

Run the released model on a studio capture of identities known to be absent from training, compare AbsRel against Sapiens-1B, and also perform an image-retrieval sweep of the 1264 test images against training data; if the gap narrows or retrieval hits appear, the reported edge is partly memorization rather than generalization.

Watch

Extended reading notes

Core claim

The core discovery, stated on the paper's own terms, is that the conditional joint distribution of facial appearance and depth can be learned end-to-end in one latent diffusion model. The forward process adds independent noise to the RGB latent and the depth latent from the same variational autoencoder, concatenates the two noisy latents into a six-channel input, and the shared UNet predicts both noises simultaneously; the reverse process therefore generates a face and its depth map from one denoising trajectory, forcing structural correlation between them. The paper reports that fine-tuning this model for masked inpainting yields state-of-the-art monocular facial depth estimation on its studio test set (AbsRel 0.162, versus 0.197 for Sapiens-1B), and that the same weights, extended with temporal and audio attention, produce audio-driven videos with consistent per-frame depth.

Load-bearing premise

The state-of-the-art depth claim rests on the assumption that the 55-identity, 1264-image studio evaluation set contains no identities also used in the 336-subject studio training set; the paper never states that separation, and both sets come from the same studio face dataset.

Editorial extensions

If this is right

  • A single set of trained weights performs both monocular facial depth estimation and depth-conditioned image generation, removing the need to cascade a separate depth network and a separate generator.
  • Audio-driven talking-head videos come out with a matching depth channel per frame, so 3D-aware editability, relighting, and compositing are available for animation that current RGB-only methods cannot provide.
  • Editing a face in the depth domain (changing nose shape, expression, or pose) regenerates a photorealistic RGB image that retains the reference identity.
  • Mixing studio scans with pseudo-ground-truth depth from in-the-wild video lets the model keep facial depth accuracy while generalizing to outdoor portraits.
  • Because appearance and depth are decoupled in one noise prediction, the generated depth is structurally tied to the generated image rather than estimated afterwards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same concatenated-latent design is a natural template for other paired modalities, such as surface normals, albedo, or segmentation, where a shared denoising pass would enforce cross-modal consistency.
  • Editorial extension: the paper's depth comparison would be more conclusive on an identity-disjoint studio test; its text does not say the 55 evaluation identities are excluded from the 336 training identities, and both come from the same studio face dataset.
  • Editorial extension: the audio-driven RGBD output could be stress-tested for temporal depth stability across frames, since a talking-head video with jittering depth would undermine the 3D uses the paper motivates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a diffusion-based framework for jointly generating portrait RGB images and depth maps. The method extends a Stable Diffusion UNet to denoise a six-channel latent formed by concatenating separately noised RGB and depth latents, conditioned by a ReferenceNet that encodes a reference identity image. Training uses studio face scans with registered-mesh depth maps and in-the-wild videos with pseudo depth from a monocular face tracker. After training, the authors fine-tune with masked inpainting for image-to-depth and depth-to-image tasks, and add audio and temporal attention for audio-driven RGBD video generation. Experiments include quantitative depth estimation against Marigold, GeoWizard, and Sapiens (Table 1), qualitative wild-depth comparisons, relighting, depth-based editing, and audio-driven animation, plus ablations for the ReferenceNet and training data.

Significance. The idea of jointly learning appearance and depth in a single diffusion pass is timely and useful; if validated, it would enable consistent RGB-D portrait generation with applications in animation, editing, and relighting. The architecture is simple and builds on established components, and the ablations provide useful evidence about the role of the ReferenceNet and the mixed training data. However, the paper's headline state-of-the-art claim for monocular facial depth estimation rests on a single evaluation whose train/test identity separation is not disclosed and which lacks error bars; the wild-depth qualitative evaluation is partly circular. These issues must be resolved before the core claim can be accepted.

major comments (3)
  1. [Section 5.1/5.2, Table 1] The paper trains on 336 subjects from the Semantic Deep Face Models studio dataset [6] (Section 5.1) and evaluates on 1264 images from 55 identities from 'an unseen studio dataset' (Section 5.2), but it never states that these 55 identities are disjoint from the 336 training subjects. Since both come from the same source and the evaluation uses the same registered-mesh depth distribution as training, an identity overlap would allow the model to memorize the target depth geometry. Please state the identity-level split or provide an image/identity overlap check; without this, the AbsRel 0.162 versus Sapiens-1B 0.197 advantage in Table 1 cannot be attributed to generalization.
  2. [Section 5.2, Table 1] The central quantitative claim is reported as single point estimates with no error bars, per-identity variance, or significance test. With 1264 images from 55 identities, the effective sample size for the comparison is at most 55; the paper should report per-identity mean and standard deviation or confidence intervals and a paired statistical test. In addition, the evaluation is masked to the facial skin region, which is exactly the region covered by the training depth maps; please justify that this masking does not favor a model trained on the same masked depth distribution.
  3. [Section 5.2, Fig. 4] The in-the-wild qualitative depth comparison is made against the output of the same monocular tracker [7, 8] that generated the pseudo ground-truth depth for the in-the-wild portion of the training data. This makes the comparison partially circular and does not substantiate the claim of better generalization to outdoor settings. A quantitative evaluation on wild images with independent ground truth (e.g., multi-view or scanned geometry) or a user study is needed to support the generalization claim.
minor comments (4)
  1. [Section 5.2, Table 1] The row 'Ours-Wild-Only' is not defined in the main text; it should be described in the ablation or experimental setup.
  2. [Equation (5)] Equation (5) contains formatting artifacts with doubled subscripts and an ambiguous norm; please rewrite it as a standard MSE loss.
  3. [Section 3.2] The update rule z_{l-1} = z_l - epsilon_theta(z_l, z_r, l) omits the noise schedule coefficients; as written it is not the usual DDPM/DDIM update, so please clarify the notation.
  4. [Related Work] The paper would benefit from a more explicit discussion of how the proposed architecture differs from previous joint diffusion models such as HyperHuman and JointNet.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central quantitative results are evaluated against scanner-derived ground truth and external baselines, and the acknowledged use of the authors' tracker for wild training labels does not force any reported prediction.

full rationale

The paper's derivation chain is self-contained. The joint training objective (Eq. 5) directly optimizes paired RGB and depth latents, and downstream tasks are implemented as masked-latent inpainting on that trained distribution, not by re-using a fitted parameter as a prediction. The main quantitative depth claim (Table 1) is evaluated against registered-mesh depth from the studio scanner [6], which is independent of the model's own outputs, and is compared against external baselines Marigold, GeoWizard, and Sapiens. The paper does not state an identity-level split between the 336 training subjects and the 55 evaluation identities, which is a data-leakage risk rather than a circular reduction. In-the-wild depth supervision uses the authors' face tracker [7,8], and the qualitative wild comparison is made against that same tracker; this weakens the evidence for wild generalization but is explicitly acknowledged in the text and does not force the reported result, because the studio ground-truth component supplies independent signal. No equation equates a fitted input with a predicted output, and no self-citation is invoked as an external authority to forbid alternatives. The limitations section further acknowledges the skin-region restriction and modest training scale, consistent with a non-circular but partially constrained evaluation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method is an empirical deep-learning system, not a derivation, so its central dependency is a large set of learned weights plus the assumptions above. No new particles, forces, or theoretical constructs are introduced. The free-parameter entry captures the entire fitted model; the axioms capture the data and architectural assumptions that the quantitative claims rest on.

free parameters (1)
  • Learned weights of the diffusion UNet, ReferenceNet, audio-attention, and temporal-attention modules = Not released; trained for 30,000 steps at learning rate 1e-5 for the joint model (Section 5.1)
    Every RGB and depth prediction is a function of these fitted weights. The paper provides no public checkpoint, so all quantitative claims depend on unreleased fitted parameters that cannot be independently audited.
assumptions (4)
  • domain assumption Latent diffusion formalism, including the noise schedule and denoising loss in Equations 1-5, remains valid when RGB and depth latents are concatenated into a single six-channel input.
    Section 3.2 generalizes the standard latent diffusion process to two concatenated latents without proving that the Stable Diffusion VAE and UNet priors transfer to depth maps. The results depend on this transfer.
  • domain assumption Pseudo ground-truth depth from the monocular face tracker [7,8] is accurate enough to supervise in-the-wild depth learning.
    Section 5.1 and Section 7.3 generate all in-the-wild training depth maps by fitting the authors' PCA face model with landmark and photometric losses. Systematic errors in this tracker become part of the training signal.
  • domain assumption The 55-identity studio evaluation set used for Table 1 is disjoint from the 336-subject studio training set.
    Section 5.2 defines the test set but never states identity-level separation from the Section 5.1 training data. The state-of-the-art depth claim depends on this unstated split.
  • domain assumption The skin-only mask applied to ground truth and predictions during depth evaluation is a fair comparison for all methods.
    Section 5.2 removes regions outside the facial skin before computing AbsRel, delta1, and RMSE. This masking may favor methods trained on face-only depth distributions and hides errors in hair, ears, and background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint Learning of Depth and Appearance for Portrait Image Animation." pith.science (2026). https://pith.science/paper/S5PJCPGD

@misc{pith2026250108649,
  author       = {Pith},
  title        = {Pith review of: Joint Learning of Depth and Appearance for Portrait Image Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5PJCPGD}},
  note         = {Machine review of arXiv:2501.08649}
}
read the original abstract

2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output.

Figures

Figures reproduced from arXiv: 2501.08649 by the authors.

Figure 1
Figure 1. The overview of the proposed pipeline. Given a ref [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The detailed architecture of the building block of our [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparisons with the state-of-the-art methods for monocular depth estimation on studio images. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons with the state-of-the-art methods for monocular depth estimation on wild faces. Note that even 3D facial [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Depth-based face editing on shape, expression and pose. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: Ablation studies to show the effect of our architecture [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The overview of our audio-driven animation network. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparisons with the state-of-the-art methods for monocular depth estimation on studio faces. We also show error [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DreamCube: 3D Panorama Generation via Multi-plane Synchronization

    cs.GR 2025-06 conditional novelty 6.0 of 10

    A synchronized multi-plane adaptation of 2D diffusion operators enables seam-consistent cubemap generation, and DreamCube extends this to joint RGB-D panorama generation and 3D scene lifting.

Reference graph

Works this paper leans on

92 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [6]

    Semantic deep face models

    Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. Semantic deep face models. In 2020 interna- tional conference on 3D vision (3DV), pages 345–354. IEEE,

    2020

  2. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 4, 1

    2020

  3. [2]

    Adabins: Depth estimation using adaptive bins

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009–4018, 2021. 2

    2021

  4. [3]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 2

    arXiv 2023

  5. [4]

    Midas v3

    Reiner Birkl, Diana Wofk, and Matthias M ¨uller. Midas v3. 1–a model zoo for robust monocular relative depth estima- tion. arXiv preprint arXiv:2307.14460, 2023. 6

    arXiv 2023

  6. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  7. [7]

    Continuous landmark detection with 3d queries

    Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, and Derek Bradley. Continuous landmark detection with 3d queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16858– 16867, 2023. 5, 7, 1

    2023

  8. [8]

    Infinite 3d landmarks: Improving continuous 2d facial landmark detection

    Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, and Derek Bradley. Infinite 3d landmarks: Improving continuous 2d facial landmark detection. In Computer Graphics Forum, page e15126. Wiley Online Library, 2024. 5, 7, 1

    2024

Show all 92 references
  1. [9]

    Lip movements generation at a glance

    Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. In Proceedings of the European conference on computer vision (ECCV), pages 520–535, 2018. 2

    2018

  2. [10]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 2, 7

    2024 arXiv

  3. [11]

    3d- aware face editing via warping-guided latent direction learn- ing

    Yuhao Cheng, Zhuo Chen, Xingyu Ren, Wenhan Zhu, Zhengqin Xu, Di Xu, Changpeng Yang, and Yichao Yan. 3d- aware face editing via warping-guided latent direction learn- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 916–926, 2024. 1

    2024

  4. [12]

    You said that? arXiv preprint arXiv:1705.02966, 2017

    Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint arXiv:1705.02966, 2017. 2

    2017 arXiv

  5. [13]

    Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation. arXiv preprint arXiv:2410.07718, 2024. 2

    arXiv

  6. [14]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 1

    2021

  7. [15]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alek- sei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022. 1

    2022

  8. [16]

    Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image. In European Conference on Computer Vision, pages 241–258. Springer, 2025. 2, 6, 7

    2025

  9. [17]

    Talk-act: Enhance textural- awareness for 2d speaking avatar reenactment with diffusion model

    Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Shengyi He, Zhiliang Xu, Haocheng Feng, Errui Ding, Jing- dong Wang, Hongtao Xie, et al. Talk-act: Enhance textural- awareness for 2d speaking avatar reenactment with diffusion model. arXiv preprint arXiv:2410.10696, 2024. 2

    arXiv

  10. [18]

    Towards zero-shot scale-aware monocu- lar depth estimation

    Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares , Ambrus,, and Adrien Gaidon. Towards zero-shot scale-aware monocu- lar depth estimation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 9233–9243,

  11. [19]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 1

    2024 arXiv

  12. [20]

    Sparsectrl: Adding sparse con- trols to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse con- trols to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023. 1

    2023 arXiv

  13. [21]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 1, 2, 5

    2023 arXiv

  14. [22]

    Spacex: Speech-driven por- trait animation with controllable expression

    Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, and Ming-Yu Liu. Spacex: Speech-driven por- trait animation with controllable expression. arXiv preprint arXiv:2211.09809, 2022. 2

    2022 arXiv

  15. [23]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  16. [24]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 1, 2

    2024

  17. [25]

    Learning high fi- delity depths of dressed humans by watching social media dance videos

    Yasamin Jafarian and Hyun Soo Park. Learning high fi- delity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12753– 12762, 2021. 2 9

    2021

  18. [26]

    Audio-driven emotional video portraits

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14080–14089, 2021. 1, 2

    2021

  19. [27]

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1–10, 2022. 2

    2022

  20. [28]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 1, 2, 4

    2024 arXiv

  21. [29]

    Mobile- portrait: Real-time one-shot neural head avatars on mobile devices

    Jianwen Jiang, Gaojie Lin, Zhengkun Rong, Chao Liang, Yongming Zhu, Jiaqi Yang, and Tianyun Zhong. Mobile- portrait: Real-time one-shot neural head avatars on mobile devices. arXiv preprint arXiv:2407.05712, 2024. 1

    2024 arXiv

  22. [30]

    Dreampose: Fashion image-to-video synthesis via stable diffusion

    Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22623–22633. IEEE, 2023. 2

    2023

  23. [31]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 1

    2019

  24. [32]

    Analyzing and improv- ing the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020. 1

    2020

  25. [33]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9492...

    2024

  26. [34]

    Sapiens: Foundation for human vision mod- els

    Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision mod- els. In European Conference on Computer Vision , pages 206–228. Springer, 2025. 2, 7

    2025

  27. [35]

    Diffu- sionclip: Text-guided diffusion models for robust image ma- nipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffu- sionclip: Text-guided diffusion models for robust image ma- nipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2426–2435,

  28. [36]

    Switchlight: Co-design of physics- driven architecture and pre-training framework for human portrait relighting

    Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switchlight: Co-design of physics- driven architecture and pre-training framework for human portrait relighting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition,...

    2024

  29. [37]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  30. [38]

    Cyberhost: Taming audio- driven avatar diffusion model with region codebook atten- tion

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Ji- aqi Yang, and Yanbo Zheng. Cyberhost: Taming audio- driven avatar diffusion model with region codebook atten- tion. arXiv preprint arXiv:2409.01876, 2024. 2

    2024 arXiv

  31. [39]

    Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding

    Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding. arXiv preprint arXiv:2405.03121, 2024. 2

    2024 arXiv

  32. [40]

    Hyperhuman: Hyper-realistic human gener- ation with latent structural diffusion

    Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realistic human gener- ation with latent structural diffusion. arXiv preprint arXiv:2310.08579, 2023. 2

    arXiv

  33. [41]

    Wonder3d: Sin- gle image to 3d using cross-domain diffusion

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Sin- gle image to 3d using cross-domain diffusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  34. [42]

    Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pag...

    2024

  35. [43]

    Dreamtalk: When expressive talking head generation meets diffusion probabilistic models

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767, 2023. 2

    2023 arXiv

  36. [44]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01900, 2024. 1

    2024 arXiv

  37. [45]

    Otavatar: One-shot talking face avatar with control- lable tri-plane rendering

    Zhiyuan Ma, Xiangyu Zhu, Guo-Jun Qi, Zhen Lei, and Lei Zhang. Otavatar: One-shot talking face avatar with control- lable tri-plane rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16901–16910, 2023. 1

    2023

  38. [46]

    Total relighting: learning to relight portraits for background replacement

    Rohit Pandey, Sergio Orts-Escolano, Chloe Legendre, Chris- tian Haene, Sofien Bouaziz, Christoph Rhemann, Paul E De- bevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4):43–1, 2021. 1

    2021

  39. [47]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  40. [48]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 666–676,

  41. [49]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  42. [50]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1

    2022 arXiv

  43. [51]

    Lite2relight: 3d-aware single image portrait relight- ing

    Pramod Rao, Gereon Fox, Abhimitra Meka, Mallikar- jun BR, Fangneng Zhan, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, et al. Lite2relight: 3d-aware single image portrait relight- ing. In ACM SIGGRAPH 2024 Conference Papers , pages 1–12, 2024. 1

    2024

  44. [52]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3, 4

    2022

  45. [53]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  46. [54]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  47. [55]

    Stylegan- xl: Scaling stylegan to large diverse datasets

    Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan- xl: Scaling stylegan to large diverse datasets. In ACM SIG- GRAPH 2022 conference proceedings, pages 1–10, 2022. 1

    2022

  48. [56]

    In- terpreting the latent space of gans for semantic face editing

    Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. In- terpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9243–9252, 2020. 1

    2020

  49. [57]

    Mvdream: Multi-view diffusion for 3d gen- eration

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d gen- eration. arXiv preprint arXiv:2308.16512, 2023. 2

    2023 arXiv

  50. [58]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 1

    2019

  51. [59]

    Ldm3d: Latent diffusion model for 3d

    Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d. arXiv preprint arXiv:2305.10853 ,

    arXiv

  52. [60]

    Synthesizing obama: learn- ing lip sync from audio

    Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1–13, 2017. 2

    2017

  53. [61]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 1, 2, 4

    2024 arXiv

  54. [62]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  55. [63]

    Realistic speech-driven facial animation with gans

    Konstantinos V ougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. Interna- tional Journal of Computer Vision, 128:1398–1413, 2020. 2

    2020

  56. [64]

    V-express: Conditional dropout for progres- sive training of portrait video generation

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progres- sive training of portrait video generation. arXiv preprint arXiv:2406.02511, 2024. 2

    2024 arXiv

  57. [65]

    Progressive disentangled representation learning for fine-grained controllable talking head synthesis

    Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17979–17989, 2023. 2

    2023

  58. [66]

    Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction

    Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021. 2

    2021 arXiv

  59. [67]

    Audio2head: Audio-driven one-shot talking- head generation with natural head motion

    Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking- head generation with natural head motion. arXiv preprint arXiv:2107.09293, 2021. 1

    2021 arXiv

  60. [68]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10039–10049,

  61. [69]

    De- signing deep networks for surface normal estimation

    Xiaolong Wang, David Fouhey, and Abhinav Gupta. De- signing deep networks for surface normal estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 539–547, 2015. 2

    2015

  62. [70]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 2, 7

    2024 arXiv

  63. [71]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. In The IEEE Con- ference on Computer Vision and Pattern Recognition Work- shops (CVPRW), 2022. 5

    2022

  64. [72]

    X-portrait: Expressive portrait anima- tion with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait anima- tion with hierarchical motion attention. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 1

    2024

  65. [73]

    Econ: Explicit clothed humans optimized via normal integration

    Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J Black. Econ: Explicit clothed humans optimized via normal integration. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 512–523, 2023. 2

    2023

  66. [74]

    Omniavatar: Geometry-guided controllable 3d head synthesis

    Hongyi Xu, Guoxian Song, Zihang Jiang, Jianfeng Zhang, Yichun Shi, Jing Liu, Wanchun Ma, Jiashi Feng, and Lin- jie Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1281...

    2023

  67. [75]

    Hallo: Hierarchical audio-driven vi- 11 sual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Li- wei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven vi- 11 sual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 1, 2, 4, 7

    2024 arXiv

  68. [76]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 ,

    arXiv

  69. [77]

    Magicanimate: Temporally consistent human im- age animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human im- age animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  70. [78]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2, 6

    2024

  71. [79]

    Megactor: Harness the power of raw video for vivid portrait animation

    Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, and Haoqiang Fan. Megactor: Harness the power of raw video for vivid portrait animation. arXiv preprint arXiv:2405.20851, 2024. 1

    2024 arXiv

  72. [80]

    Gaussiantalker: Speaker-specific talking head synthesis via 3d gaussian splatting

    Hongyun Yu, Zhan Qu, Qihang Yu, Jianchuan Chen, Zhonghua Jiang, Zhiwen Chen, Shengyu Zhang, Jimin Xu, Fei Wu, Chengfei Lv, et al. Gaussiantalker: Speaker-specific talking head synthesis via 3d gaussian splatting. In Proceed- ings of the 32nd ACM International Conference on Mul...

    2024

  73. [81]

    Chatface: Chat-guided real face edit- ing via diffusion latent space manipulation

    Dongxu Yue, Qin Guo, Munan Ning, Jiaxi Cui, Yuesheng Zhu, and Li Yuan. Chatface: Chat-guided real face edit- ing via diffusion latent space manipulation. arXiv preprint arXiv:2305.14742, 2023. 1

    2023 arXiv

  74. [82]

    Unsupervised high-resolution portrait gaze correction and animation.IEEE Transactions on Image Processing, 31:5272–5286, 2022

    Jichao Zhang, Jingjing Chen, Hao Tang, Enver Sangineto, Peng Wu, Yan Yan, Nicu Sebe, and Wei Wang. Unsupervised high-resolution portrait gaze correction and animation.IEEE Transactions on Image Processing, 31:5272–5286, 2022. 1

    2022

  75. [83]

    Joint- net: Extending text-to-image diffusion for dense distribution modeling

    Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, and Yao Yao. Joint- net: Extending text-to-image diffusion for dense distribution modeling. arXiv preprint arXiv:2310.06347, 2023. 2

    2023 arXiv

  76. [84]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3

    2023

  77. [85]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  78. [86]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 5

    2021

  79. [87]

    Identity- preserving talking face generation with landmark and ap- pearance priors

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity- preserving talking face generation with landmark and ap- pearance priors. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 9729–...

    2023

  80. [88]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186,

  81. [89]

    Makelttalk: speaker-aware talking-head animation

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6):1–15, 2020. 2

    2020

  82. [90]

    Vlogger: Make your dream a vlog

    Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream a vlog. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8806–8817, 2024. 1 12 Joint Learning of Depth and Appearanc...

    2024

  83. [91]

    Implementation Details We describe more implementation details on the network architecture and the training details used in Section 3 of the main paper. 7.1. Network Architecture Here, we present the details of our network architecture for audio-driven animation (Sec. 4.2), as...

  84. [92]

    Depth Estimation As demonstrated in Sec

    More Results 8.1. Depth Estimation As demonstrated in Sec. 5.2, we apply a mask on the face skin region while calculating the depth estimation metrics with ground truth. In Fig. 10, we show more qualitative re- sults along with the error map under the mask. Here we set the err...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.