Pith. sign in

REVIEW 3 major objections 4 minor 55 references

IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a video diffusion model trained only on monocular clips can generate talking-head videos as Multiplane Images, so novel views render in one pass without reconstruction.

desk verdict Single-pass MPI video diffusion for talking heads is a real advance and the paper deserves a serious referee, but the bootstrapped geometry supervision is under-validated and the missing code/data makes the quantitative claims hard to trust. read the letter →

arxiv 2504.19165 v2 pith:B4BKXC7D submitted 2025-04-27 cs.CV

Yuan Li , Ziqian Bai , Feitong Tan , Zhaopeng Cui , Sean Fanello , Yinda Zhang This is my paper · ORCID
classification cs.CV
keywords talkingheadsynthesisvideodiffusionmodelsmultiplaneimagesmonoculartraining3D-awaregenerationnovelviewfacereenactmentVRstereorendering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single video diffusion model, trained only on monocular in-the-wild videos, can generate talking-head videos that are already 3D: the output is a Multiplane Image (MPI) video, so stereo pairs and perspective shifts are rendered by one differentiable compositing pass instead of a separate NeRF or Gaussian reconstruction. This matters because existing diffusion-based talking-head methods are 2D and need extra 3D lifting, while 3D avatar methods do not inherit diffusion's image quality. The paper's load-bearing solution is reference-target alternating training, in which the model sometimes builds the MPI in the target camera (supervised by the true frame) and sometimes in the reference camera (supervised by its own bootstrapped pseudo ground truth under late-stage noise), letting one network learn sharp appearance and view-consistent geometry at once. The authors report competitive image and video quality against 2D diffusion baselines and novel-view rendering comparable to a NeRF-based one-shot avatar baseline, at substantially faster render speed.

What carries the argument

The load-bearing object is the MPI video, defined as a sequence of D RGBA planes parallel to a frontal camera, equally spaced in disparity, where rendering from any nearby camera is a homography warp plus $\alpha$ blend. The argument is carried by a video diffusion U-Net that outputs an MPI video in the reverse process: the network predicts MPI frames, a differentiable renderer turns them into the estimated clean frontal video, and that rendered video plugs back into the denoising update. The color branch predicts frontal and residual RGB, the geometry branch predicts per-plane $\alpha$, and zero-convolution lets information flow from color to geometry so the appearance branch can bootstrap the shape branch. Training alternates two MPI camera placements, the target camera (ground-truth images, 80% of iterations) and the reference camera (pseudo ground truth from the bootstrapped model, 20% of iterations), with late-stage noise sampling $t \in [T/2, T]$ suppressing the pseudo image's flaws. This mechanism is what converts monocular head-pose variation into weak multi-view supervision, and it is also what makes the whole pipeline a single denoising run.

What would settle it

Using synchronized multi-camera footage of a subject, render side views from the MPIs at increasing baselines and compare MPI-derived disparity with ground-truth disparity; if disparity error grows sharply as the baseline increases while frontal quality stays high, the reference-camera bootstrap is encoding appearance bias rather than true geometry. A cheaper check on public data is to compare the MPI disparity in rigid facial regions such as the forehead and nose with the fitted parametric-model depth at matched head poses and see whether disagreement falls inside the render tolerance.

Watch

Extended reading notes

Core claim

The paper's central claim is that 3D-aware talking-head video can be generated as Multiplane Image (MPI) video in a single denoising pass from monocular training data. Each generated frame is an MPI: a stack of RGBA planes parallel to a chosen frontal camera; warping those planes by homographies and alpha-compositing them yields the image in any nearby camera. The model is conditioned on one identity image and expression and head-pose controls from a parametric head model, and its network has a color branch (predicting frontal and residual RGB) and a geometry branch (predicting per-plane alpha), coupled by zero-convolution. Training on ordinary monocular videos alternates between building the MPI in the target camera, where the ground-truth frame supervises sharp appearance, and in the reference camera, where the target frame serves as a side-view ground truth for geometry; the reference-camera branch is fed a pseudo ground truth generated by the bootstrapped model itself, with noise sampled only from the late half of the diffusion schedule to hide its imperfections. The result is that novel-view rendering, including stereo pairs and camera push-ins, is a direct render call on the denoised MPI video rather than a separate reconstruction or optimization step.

Load-bearing premise

The 3D training signal is the model's own rendered image in the reference camera, so if that bootstrapped image is structurally wrong the learned shape inherits the same error, and the authors concede in the supplement that these pseudo images can drift in color tone.

Editorial extensions

If this is right

  • Novel stereo rendering becomes a byproduct: after one denoising run, binocular or shifted-camera views are produced by alpha-compositing the MPI planes, with no per-view sampling or 3D reconstruction.
  • Training needs only ordinary monocular talking-head videos; the alternating reference and target camera scheme supplies weak multi-view supervision from head-pose variation, so the method can scale to large in-the-wild collections.
  • At inference the paper reports roughly 109 FPS at 512x512 for novel-view rendering of its MPI pipeline, about an order of magnitude faster than the NeRF-based Portrait4D-v2 baseline, while still producing visually comparable stereo disparity.
  • Long videos are stitched autoregressively: each clip feeds its last generated frame to the next clip as a first-frame condition, with a scheduled guidance scale that prevents error accumulation across clips.
  • Because MPI planes can be composited by a differentiable renderer, the same network can be trained end-to-end without ground-truth multi-view video, which is the main practical enabler of the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim transfers beyond faces, the alternating-camera bootstrap is a general recipe: any monocular video with known relative camera pose between frames could train a single-pass MPI diffusion model for objects, hands, or bodies; this is an inference, not a paper claim.
  • The paper's own limitation statement about excessive viewpoint changes implies the practical promise is modest-baseline VR; a natural next test would be the same denoising-with-rendering loop on a 3D Gaussian or mesh representation to see whether the view range is bounded by MPI or by the training scheme.
  • A tension the paper does not resolve is that the pseudo ground truth used to supervise geometry is generated by the same appearance model whose errors it is supposed to correct; in practice this makes the claimed 3D consistency an upper bound on the bootstrap's structural accuracy, and a small multi-view validation set would expose how tight that bound is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes IM-Portrait, a video diffusion model that generates talking-head videos as Multiplane Image (MPI) videos in a single denoising pass, conditioned on a reference identity image and 3DMM-based expression/pose controls. The key architectural claim is that the model outputs an MPI sequence directly, so novel views (stereo pairs, perspective shifts) can be rendered by MPI compositing without a separate 3D reconstruction stage, NeRF fitting, or Gaussian splatting. To train from monocular in-the-wild videos, the authors introduce an alternating reference-target training scheme: when MPIs are constructed in the target camera, the target frame supervises the frontal view (Eq. 6); when MPIs are constructed in the reference camera, the model bootstraps its own pseudo ground truth and supervises a side view (Eq. 7), which is the only explicit multi-view geometric supervision. Experiments compare against 2D talking-head methods, a NeRF-based baseline, and a 2D-diffusion-plus-depth baseline, reporting competitive image/video metrics, the best FVD on several datasets, efficient rendering, and qualitative stereo/perspective results.

Significance. If validated, this is a useful step toward 3D-aware generative video: directly producing a renderable 3D representation from a single diffusion process, without multi-view training data or post-hoc reconstruction, is a clean and potentially practical formulation for VR/AR talking-head applications. The paper reports broad experiments across multiple datasets, several baselines, an ablation of the residual image, an ablation of late-stage noise sampling, and a comparison against a NeRF-based 3D baseline, and it ships a large body of qualitative evidence including side views and disparity maps. The rendering speed advantage (109 FPS at 512x512) is concrete and well aligned with the stated application. However, the load-bearing geometric claim rests on a self-supervised bootstrap whose validity is not independently established, and the evaluation does not measure geometric accuracy against ground-truth novel views or known geometry, so the 3D-consistency claim currently remains plausible but not fully verified.

major comments (3)
  1. [Sec. 3.3, Eq. (7), Fig. 3; Supp. Sec. F] The only multi-view supervision for geometry is L_side in Eq. (7), where the side-view ground truth is the image O^mpi_k rendered from the MPIs generated by the model itself in the reference camera. This is a self-training bootstrap, not independent supervision. The authors acknowledge in Supp. Sec. F that the pseudo ground truth images exhibit "drifting in the color tone," and the late-stage noise sampling t in [T/2,T] only down-weights, rather than removes, the mismatch at the smallest t that is actually used. Because the same network produces both the color and geometry branches, systematic color-branch errors can enter the geometry branch's training target, so the reported 3D consistency may partly reflect the model's internal bias rather than true shape. The paper should provide a concrete test that separates these: for example, a held-out multiview or known-geometry experiment (e.g., render MPIs from a known mesh and compare disparity or depth error), or an ablation in which L_side is supervised by real multiview frames instead of pseudo ground truth on a small dataset, quantifying how much geometry quality degrades.
  2. [Sec. 4.2, Sec. G, Table D, Fig. 5] The reported novel-view evaluation does not measure geometric accuracy against ground truth. The stereo disparity maps in Fig. 5 are estimated from the model's own rendered images with RAFT-Stereo, so they can look plausible even if the depth is a smooth prior rather than correct geometry. The side-view FID in Table D compares distributions of generated images, not the correctness of the rendered geometry. The paper claims parity with the NeRF-based Portrait4D-v2 on 3D effects, but no metric on depth, disparity, or novel-view PSNR/SSIM against ground-truth images of a known head model or a multiview dataset is reported. I ask the authors to add a quantitative geometry metric (e.g., disparity error on a synthetic or lightstage-like sequence, or novel-view PSNR/SSIM against a held-out multidataset) and to report such a metric also for the ablations in Sec. 4.3, where the claim that the residual image improves geometry is currently supported mainly by the qualitative Fig. 6.
  3. [Table 1, Table A, Supp. Sec. C] Several quantitative claims are made without error bars or significance tests. For example, in Table 1 the HDTF LPIPS difference between Ours (0.118) and X-Portrait (0.119) is within any reasonable run-to-run variation, and in Table A the FID differences between Ours and X-Portrait on VFHQ (33.10 vs. 26.22) are opposite in sign to the HDTF ranking. The central claim 'best FVD overall' should be supported by variance estimates across evaluation subsets or multiple training runs, and the authors should state how many evaluation clips and frames each metric uses, especially for the self-collected dataset (50 identities, 32 frames each). This is a standard reproducibility requirement and would strengthen the robustness of the performance comparisons.
minor comments (4)
  1. [Throughout] Please fix typos and notation inconsistencies: 'pesudo' (Sec. 3.3), 'specifiv' (Supp. Sec. A), 'X-xportrait' (Sec. 4.2), and the inconsistent naming 'Talkinghead1kh' (main paper) vs 'Talkinghead1Kh' (elsewhere). Also, the parameter 'r' for near/far plane placement in Supp. Sec. A should be defined precisely (distance from MPI frontal camera to which 3DMM head joint?).
  2. [Sec. 3.3] The equation environment for Eq. (7) is not typeset correctly (the L_side definition appears inline as 'Lside θ = EOref,Ompi,Cmpi,t,Oside,ϕside[∥Oside− ˆOside t−1∥2]'), and it is not consistent with the LaTeX formatting of Eq. (6). Please ensure both equations are rendered with the same style.
  3. [Sec. 4 and Supp. Sec. A] Since the method is trained on a self-collected 35k-video dataset that is not released, and the code is not released, the paper should include an explicit reproducibility statement describing the data collection and any plans to release code or pretrained models.
  4. [Fig. 3] The description of the top and bottom branches in the figure caption is helpful, but the caption should clarify which branch corresponds to Eq. (6) and which to Eq. (7) in the main text to make the connection immediate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predicted MPIs are supervised by real monocular target frames, and the pseudo-GT bootstrap is an input-alignment trick, not a self-defined prediction.

full rationale

The paper's derivation chain is not circular. The diffusion model g_theta predicts an MPI video S, which is rendered to 2D observations via the MPI renderer R (Eqs. 4-5). Training uses two losses: Eq. 6 compares the frontal render of an MPI built in the target camera with the real target frame O_mpi, and Eq. 7 compares a render from an MPI built in the reference camera with the real target frame O_side. In both cases the supervision target is a real monocular video frame, not the model's own output. The bootstrapping mechanism in Sec. 3.3 generates the model input under the reference camera by rendering a previously sampled MPI (\hat{O}^{mpi}_k) and adding noise; this pseudo GT is an input-alignment device, not the quantity being predicted. Eq. 7 still penalizes the distance to the real target frame, so the novel-view prediction does not reduce to the pseudo GT by construction. The paper explicitly acknowledges the pseudo images need only have correct global structure and admits color-tone drift (Supp. Sec. F), which is a training-quality caveat rather than a circular step. The architecture is built on external prior work (Lumiere [5], Tewari et al. [37], Tucker and Snavely [39], Zhao et al. [53]), and the authors' own prior publications [3,4,44] appear only in related-work context and are not load-bearing for the central claim. Evaluations are against external benchmarks and real frames (HDTF, Talkinghead1kh, VFHQ, self-collected data; FID/FVD/L1/LPIPS), and novel-view quality is measured by FID under randomized viewpoints (Table D). No equation equates the predicted output to a fitted parameter or to a self-citation chain, so the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method does not postulate new physical entities or forces. It introduces a new training signal (pseudo ground truth) and a new way to combine known components (diffusion, MPIs, 3DMM), but these are not 'invented entities' in the sense of unobserved particles or mediators. The main assumptions are about the sufficiency of MPI and the validity of self-supervision.

free parameters (5)
  • Late-stage noise sampling range = t in [T/2, T]
    Chosen by hand to hide the distribution mismatch of pseudo GT; the paper shows that full noise sampling leads to blurriness (Sec. 3.3, ablation in Sec. 4.3).
  • Classifier-free guidance scales = reference: 1.5, first-frame: linear 1.0 to 0.0 over frames 1-16, then 0.0
    Set manually; the supplementary shows a constant scale causes artifacts, so the schedule is a hand-tuned hyperparameter.
  • Loss selection probability = 0.8 for L_mpi, 0.2 for L_side
    Chosen by hand to balance image detail learning (target camera) and geometry learning (reference camera).
  • Loss weights = LPIPS 0.1, mask 0.01, depth smoothing 0.01, disparity 0.001
    Standard multi-loss balancing; no sensitivity analysis is provided.
  • MPI near/far plane offsets = near = r - 0.15, far = r + 0.05
    Set based on distance r from the camera to the head joint; chosen to cover the head extent.
assumptions (5)
  • domain assumption Multiplane Image representation is a sufficient 3D representation for talking head novel-view rendering within a limited viewpoint range.
    Sec. 3.1 and Sec. 5 acknowledge that extreme viewpoint changes are not supported, but the paper assumes MPI is adequate for the evaluated stereo and perspective effects.
  • domain assumption Monocular video with natural head pose variation provides enough multi-view supervision to learn 3D geometry.
    Sec. 3.3 justifies the reference-target alternating training on this premise; without it, the side-view loss has no supervision.
  • domain assumption 3DMM fitting yields accurate camera poses and expression coefficients for constructing MPIs in the reference and target camera spaces.
    The whole pipeline (Sec. 3.2, Sec. 4 implementation) depends on 3DMM fits to rasterize UV coordinate maps and align the MPI frontal camera.
  • ad hoc to paper The bootstrapped pseudo ground truth images generated by the model in the reference camera are a valid training signal for the geometry branch.
    The paper introduces this in Sec. 3.3; it is a self-supervised loop with no external verification that the pseudo GT is geometrically correct.
  • standard math The forward diffusion model and MPI rendering are differentiable and correctly integrated into the denoising process.
    The DDPM formulation (Sec. 3.1) and the rendering operator R (Sec. 3.2) are standard; the paper relies on these as known-working components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos." pith.science (2026). https://pith.science/paper/B4BKXC7D

@misc{pith2026250419165,
  author       = {Pith},
  title        = {Pith review of: IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4BKXC7D}},
  note         = {Machine review of arXiv:2504.19165}
}
read the original abstract

We propose a novel 3D-aware diffusion-based method for generating photorealistic talking head videos directly from a single identity image and explicit control signals (e.g., expressions). Our method generates Multiplane Images (MPIs) that ensure geometric consistency, making them ideal for immersive viewing experiences like binocular videos for VR headsets. Unlike existing methods that often require a separate stage or joint optimization to reconstruct a 3D representation (such as NeRF or 3D Gaussians), our approach directly generates the final output through a single denoising process, eliminating the need for post-processing steps to render novel views efficiently. To effectively learn from monocular videos, we introduce a training mechanism that reconstructs the output MPI randomly in either the target or the reference camera space. This approach enables the model to simultaneously learn sharp image details and underlying 3D information. Extensive experiments demonstrate the effectiveness of our method, which achieves competitive avatar quality and novel-view rendering capabilities, even without explicit 3D reconstruction or high-quality multi-view training data.

Figures

Figures reproduced from arXiv: 2504.19165 by the authors.

Figure 1
Figure 1. We propose a 3D-aware video diffusion model for talking head synthesis. Given an image as identity and a sequence of tracking [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Inference pipeline. Our model is built on the architecture of Lumiere [5], which takes an identity image, 2D noise video, the sequence of expressions rendered from 3DMM model and first frame image as input and outputs MPI video sequences. During inference, the network is conditioned on a reference portrait and takes the last frame of previously generated clip as the first frame condition. We separate the network int… view at source ↗
Figure 3
Figure 3. Illustration of Reference-Target Alternating Training. On the top branch, we construct MPIs in the target camera and render the frontal view, where we directly use the ground truth target image to compute loss Eq. (6) to learn sharp renderings and bootstrap the model. On the bottom branch, we construct MPIs in the reference camera. We first rasterize the target control signal (expression) in the reference camera, th… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Talking head results. We show results from previous work and our method. Our method generates results with sharp appearance and closely aligned expression with the ground truth. Note Portrait4d-v2 uses a customized camera space which is non-trivial to render GT camera …
Figure 5
Figure 5. Figure 5: Binocular stereo and perspective effects. Our method generates MPI videos that enables efficient spatial rendering, e.g. binocular stereo and perspective effects. We render stereo pairs and calculate a disparity using RAFT-Stereo [29] to visualize the perceived geometr…
Figure 7
Figure 7. Figure 7: Effect of Late-stage noise sampling. We show a com￾parison between our model trained with and without the Late-stage noise sampling. Training with regular noise sampling leads to ex￾cessive blurriness and lost of details. 5. Conclusion We present a method to learn a 3D…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 28 canonical work pages

  1. [1]

    Ren- derdiffusion: Image diffusion for 3d reconstruction, inpaint- ing and generation

    Titas Anciukevi ˇcius, Zexiang Xu, Matthew Fisher, Paul Hen- derson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Ren- derdiffusion: Image diffusion for 3d reconstruction, inpaint- ing and generation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 12608–12618, 2023. 3, 4

    work page 2023

  2. [2]

    Rignerf: Fully controllable neu- ral 3d portraits

    ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neu- ral 3d portraits. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 20364– 20373, 2022. 3

    work page 2022

  3. [3]

    Learning personal- ized high quality volumetric head avatars from monocular rgb videos

    Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Ming- song Dou, Sergio Orts-Escolano, et al. Learning personal- ized high quality volumetric head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16890– 16900, 2023

    work page 2023

  4. [4]

    Effi- cient 3d implicit head avatar with mesh-anchored hash table blendshapes

    Ziqian Bai, Feitong Tan, Sean Fanello, Rohit Pandey, Ming- song Dou, Shichen Liu, Ping Tan, and Yinda Zhang. Effi- cient 3d implicit head avatar with mesh-anchored hash table blendshapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1975– 1984, 2024. 3

    work page 1975

  5. [5]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 4, 6, 1

    arXiv 2024

  6. [6]

    A morphable model for the synthesis of 3d faces

    V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157–164. 2023. 4

    work page 2023

  7. [7]

    Hyperreenact: One-shot reenactment via jointly learning to refine and re- target faces

    Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, and Georgios Tzimiropoulos. Hyperreenact: One-shot reenactment via jointly learning to refine and re- target faces. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2

    work page 2023

  8. [8]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 6, 1

    work page 2023

Show all 55 references
  1. [9]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion

    Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mo- hammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion. In ICML, 2024. 2

    2024

  2. [10]

    Personalized face modeling for improved face reconstruction and motion retargeting

    Bindita Chaudhuri, Noranart Vesdapunt, Linda Shapiro, and Baoyuan Wang. Personalized face modeling for improved face reconstruction and motion retargeting. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16 , pages 1...

    2020

  3. [11]

    Monogaus- sianavatar: Monocular gaussian point-based head avatar

    Yufan Chen, Lizhen Wang, Qijing Li, Hongjiang Xiao, Shengping Zhang, Hongxun Yao, and Yebin Liu. Monogaus- sianavatar: Monocular gaussian point-based head avatar. In ACM SIGGRAPH 2024 Conference Papers, pages 1–9, 2024. 3

    2024

  4. [12]

    Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer

    Yu Deng, Duomin Wang, and Baoyuan Wang. Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer. arXiv preprint arXiv:2403.13570, 2024. 3, 6

    2024 arXiv

  5. [13]

    Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer

    Yu Deng, Duomin Wang, and baoyuan Wang. Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer. arXiv, 2024. 2, 3, 6, 7, 4

    2024

  6. [14]

    Diffusionrig: Learning personalized priors for facial appearance editing

    Zheng Ding, Xuaner Zhang, Zhihao Xia, Lars Jebe, Zhuowen Tu, and Xiuming Zhang. Diffusionrig: Learning personalized priors for facial appearance editing. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12736–12746, 2023. 2

    2023

  7. [15]

    Headgan: One-shot neural head synthesis and editing

    Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In Proceedings of the IEEE/CVF International con- ference on Computer Vision, pages 14398–14407, 2021. 2

    2021

  8. [16]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alek- sei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022. 2

    2022

  9. [17]

    Emoportraits: Emotion-enhanced multimodal one-shot head avatars

    Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos V ougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pan- tic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , page...

    2024

  10. [18]

    From data to functa: Your data point is a function and you can treat it like one

    Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204, 2022. 3

    2022 arXiv

  11. [19]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on mach...

  12. [20]

    Dynamic neural radiance fields for monocular 4d facial avatar reconstruction

    Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8649–8658, 2021. 3

    2021

  13. [21]

    Reconstructing personalized se- mantic facial nerf models from monocular video

    Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. Reconstructing personalized se- mantic facial nerf models from monocular video. ACM Transactions on Graphics (TOG), 41(6):1–12, 2022. 3

    2022

  14. [22]

    Neural head avatars from monocular rgb videos

    Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18653–18664, 2022. 2

    2022

  15. [23]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 2

    2024 arXiv

  16. [24]

    Gans trained by a 9 two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a 9 two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6

    2017

  17. [25]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  18. [26]

    Shap-e: Generat- ing conditional 3d implicit functions

    Heewoo Jun and Alex Nichol. Shap-e: Generat- ing conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023. 3

    2023 arXiv

  19. [27]

    Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation

    Yushi Lan, Fangzhou Hong, Shuai Yang, Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation. In European Conference on Computer Vision , pages 112–130. Springer, 2025. 3

    2025

  20. [28]

    Learning a model of facial shape and expression from 4d scans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194–1, 2017. 4

    2017

  21. [29]

    Raft-stereo: Multilevel recurrent field transforms for stereo matching

    Lahav Lipson, Zachary Teed, and Jia Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching. In 2021 International Conference on 3D Vision (3DV) , pages 218–227. IEEE, 2021. 7, 8

    2021

  22. [30]

    Diffdub: Person-generic visual dubbing using inpaint- ing renderer with diffusion auto-encoder

    Tao Liu, Chenpeng Du, Shuai Fan, Feilong Chen, and Kai Yu. Diffdub: Person-generic visual dubbing using inpaint- ing renderer with diffusion auto-encoder. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3630–3634. IEEE,

    2024

  23. [31]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01900, 2024. 2, 4, 6, 7, 1

    2024 arXiv

  24. [32]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 2

    2021

  25. [33]

    Diffrf: Rendering-guided 3d radiance field diffusion

    Norman M ¨uller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328–4338, 2023. 3

    2023

  26. [34]

    Gaus- sianavatars: Photorealistic head avatars with rigged 3d gaus- sians

    Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaus- sianavatars: Photorealistic head avatars with rigged 3d gaus- sians. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 2...

  27. [35]

    Relightable gaussian codec avatars

    Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 130–141, 2024. 3

    2024

  28. [36]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 2

    2019

  29. [37]

    Diffusion with forward models: Solv- ing stochastic inverse problems without direct supervision

    Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Fr ´edo Durand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solv- ing stochastic inverse problems without direct supervision. Advances in Neural Information Processing System...

    2023

  30. [38]

    Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano

    Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. Real- time radiance fields for single-image portrait view synthesis. In SIGGRAPH, 2023. 4

    2023

  31. [39]

    Single-view view synthe- sis with multiplane images

    Richard Tucker and Noah Snavely. Single-view view synthe- sis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 551–560, 2020. 3, 5

    2020

  32. [40]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  33. [41]

    Lion: Latent point dif- fusion models for 3d shape generation

    Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point dif- fusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35:10021–10039, 2022. 3

    2022

  34. [42]

    Rodin: A generative model for sculpting 3d digital avatars using diffusion

    Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and...

    2023

  35. [43]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10039–10049,

  36. [44]

    Mvdd: Multi-view depth diffusion mod- els

    Zhen Wang, Qiangeng Xu, Feitong Tan, Menglei Chai, Shichen Liu, Rohit Pandey, Sean Fanello, Achuta Kadambi, and Yinda Zhang. Mvdd: Multi-view depth diffusion mod- els. In European Conference on Computer Vision , pages 236–253. Springer, 2025. 3

    2025

  37. [45]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. In The IEEE Con- ference on Computer Vision and Pattern Recognition Work- shops (CVPRW), 2022. 2

    2022

  38. [46]

    X-portrait: Expressive portrait anima- tion with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait anima- tion with hierarchical motion attention. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 2, 4, 6, 7, 1

    2024

  39. [47]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv:2406.09414, 2024. 4

    2024 arXiv

  40. [48]

    Styleheat: One-shot high-resolution ed- itable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution ed- itable talking face generation via pre-trained stylegan. In European conference on computer vision , pages 85–101. ...

    2022

  41. [49]

    Fast bi-layer neural synthesis of one- shot realistic head avatars

    Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one- shot realistic head avatars. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16 , pages 524–540....

    2020

  42. [50]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 4

    2023

  43. [51]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  44. [52]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 6

    2021

  45. [53]

    Generative multi- plane images: Making a 2d gan 3d-aware

    Xiaoming Zhao, Fangchang Ma, David G ¨uera, Zhile Ren, Alexander G Schwing, and Alex Colburn. Generative multi- plane images: Making a 2d gan 3d-aware. In European con- ference on computer vision, pages 18–35. Springer, 2022. 1

    2022

  46. [54]

    2D diffu- sion + depth

    Wojciech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4574–4584, 2023. 3 11 IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Head...

    2023

  47. [55]

    In our experiments, all the side view render- ings, stereo renderings and rendering speed measurements are conducted through the first method

    Rasterizing the 3DMM UV coordinate maps into freely selected MPI frontal cameras as the diffusion controlling signals{C}. In our experiments, all the side view render- ings, stereo renderings and rendering speed measurements are conducted through the first method. When we rend...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.