One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model
Pith reviewed 2026-06-26 17:52 UTC · model grok-4.3
The pith
A conditional diffusion model uses 3D normal maps and color prompts from one human image to generate novel poses and views including occluded parts.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Conditioning a denoising diffusion model on 3D normal maps and color prompts extracted from a single reference image enables high-quality synthesis of arbitrary target poses and novel views, including accurate recovery of occluded or invisible body parts through iterative conditional steps.
What carries the argument
3D prior guided diffusion model that conditions each denoising step on a 3D normal map for geometry and a color prompt for appearance.
If this is right
- Synthesis of complex and arbitrary poses succeeds without dependence on ambiguous 2D keypoints.
- Occluded and invisible human parts are recovered more accurately than in keypoint-based or NeRF-based baselines.
- A self-reconstruction refinement step produces finer details when the model is applied to previously unseen persons.
- Performance and generalization improve across multiple public human image datasets relative to earlier methods.
Where Pith is reading between the lines
- The same conditioning strategy could be tested on non-human categories if comparable 3D priors can be obtained from single images.
- Because the generation unfolds over many denoising steps, the framework may support progressive editing or multi-view consistency constraints without retraining.
- Errors in the initial 3D prior extraction could be mitigated by the diffusion model's ability to sample plausible completions, suggesting robustness to imperfect geometry inputs.
Load-bearing premise
Reliable 3D normal maps and color prompts can be extracted from a single reference human image to serve as effective conditions for the diffusion process across arbitrary poses and occlusions.
What would settle it
A controlled test set of reference images where the extracted 3D normal maps are intentionally perturbed or incomplete, followed by measurement of whether generated occluded regions still match ground-truth target images.
Figures
read the original abstract
This paper addresses the challenge of one-shot novel view and pose human image synthesis. The existing methods transfer the reference human image to a target pose using a set of 2D pose keypoints or synthesize human images based on generalizable human NeRF which uses human model priors to extract point-wise features. However, pose transfer based methods can not handle complex human pose using ambiguous 2D pose as the condition, while generalizable human NeRFs may be inaccurate to recover occluded/invisiable human parts without extracted reliable features. To solve these problems, we propose a novel approach for novel view and pose synthesis from a singe human image via conditional denoising diffusion model. Our diffusion model divides the novel view and pose synthesis problem into a sequence of conditional denoising steps. Specifically, to generate humans with complex and arbitrary poses, we introduce 3D human priors, i.e., 3D normal map and color prompt, as geometry and color conditions into the generation process. By transferring the reference human into the target human with a series of diffusion steps, our diffusion model enables high-quality synthesis including the occluded/invisible parts. Further, we propose a self-reconstruction based customized refinement to enhance fine details when tested on novel persons.Experimental results on different public datasets demonstrate that our approach significantly outperforms previous methods and also shows better generalization ability across datasets. The code will be made publicly available at https://github.com/Yankeegsj/3DPGDM.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a conditional denoising diffusion model for one-shot novel view and pose human image synthesis from a single reference image. It divides the task into sequential denoising steps conditioned on 3D human priors (normal maps and color prompts) for geometry and appearance, aiming to handle complex/arbitrary poses and occlusions better than 2D-pose transfer or generalizable NeRF methods; a self-reconstruction refinement step is added for novel persons, with claims of superior performance and cross-dataset generalization on public benchmarks.
Significance. If the empirical claims hold and the 3D-prior extraction proves robust, the work would offer a practical advance in conditional diffusion for human synthesis by explicitly injecting 3D geometry cues, potentially improving handling of invisible regions over purely 2D or implicit NeRF baselines. The promised public code release would support reproducibility.
major comments (2)
- [Abstract] Abstract: the central claim that 3D normal maps and color prompts extracted from one reference image serve as reliable conditions for arbitrary target poses and occlusions is load-bearing, yet the text provides no description of the extraction/fitting procedure, no ablation on its accuracy for occluded regions, and no preliminary validation that errors in the prior do not propagate through denoising.
- [Abstract] Abstract: the assertion that the method 'significantly outperforms previous methods' and shows 'better generalization ability across datasets' is presented without any quantitative tables, metrics, baselines, or experimental setup details, making it impossible to assess whether the reported superiority is supported by the data.
minor comments (1)
- [Abstract] Abstract: the phrasing 'transferring the reference human into the target human with a series of diffusion steps' is vague; clarify how the reference image is injected beyond the 3D priors.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We address each major comment below and indicate planned revisions to strengthen the abstract.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central claim that 3D normal maps and color prompts extracted from one reference image serve as reliable conditions for arbitrary target poses and occlusions is load-bearing, yet the text provides no description of the extraction/fitting procedure, no ablation on its accuracy for occluded regions, and no preliminary validation that errors in the prior do not propagate through denoising.
Authors: We agree the abstract is a high-level summary and lacks these specifics. The extraction/fitting procedure (SMPL-based normal map generation and reference color prompting) is detailed in the Methods section. To address the concern directly in the abstract, we will add a concise clause describing the prior extraction and note that experiments validate robustness to prior inaccuracies in occluded areas. This will be incorporated in the revised version. revision: yes
-
Referee: [Abstract] Abstract: the assertion that the method 'significantly outperforms previous methods' and shows 'better generalization ability across datasets' is presented without any quantitative tables, metrics, baselines, or experimental setup details, making it impossible to assess whether the reported superiority is supported by the data.
Authors: Abstracts conventionally summarize outcomes at a high level without tables or full experimental protocols due to length constraints. The supporting quantitative results (metrics, baselines including 2D pose transfer and generalizable NeRF methods, cross-dataset evaluation) appear in the Experiments section. We will revise the abstract to include a brief qualifier such as 'as shown in our experiments' to better link the claim to the reported data. revision: partial
Circularity Check
No significant circularity; empirical claims with no derivations
full rationale
The manuscript text (abstract and description) presents a conditional diffusion model using 3D normal maps and color prompts extracted from a single reference image, along with a self-reconstruction refinement step, but contains no equations, mathematical derivations, parameter-fitting procedures, or load-bearing self-citations. The central claim of outperforming prior methods rests entirely on experimental results across public datasets rather than any self-referential definitions, fitted inputs renamed as predictions, or ansatzes smuggled via citation. No steps reduce by construction to the paper's own inputs.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Neural human performer: Learning generalizable radiance fields for human performance rendering,
Y . Kwon, D. Kim, D. Ceylan, and H. Fuchs, “Neural human performer: Learning generalizable radiance fields for human performance rendering,” inNeurIPS, 2021, pp. 24 741–24 752
2021
-
[2]
Mps-nerf: Generalizable 3d human rendering from multiview images,
X. Gao, J. Yang, J. Kim, S. Peng, Z. Liu, and X. Tong, “Mps-nerf: Generalizable 3d human rendering from multiview images,”IEEE TPAMI, 2022
2022
-
[3]
Mononhr: Monocular neural human renderer,
H. Choi, G. Moon, M. Armando, V . Leroy, K. M. Lee, and G. Rogez, “Mononhr: Monocular neural human renderer,” in3DV. IEEE, 2022, pp. 242–251
2022
-
[4]
Sherf: Generalizable human nerf from a single image,
S. Hu, F. Hong, L. Pan, H. Mei, L. Yang, and Z. Liu, “Sherf: Generalizable human nerf from a single image,”arXiv preprint arXiv:2303.12791, 2023
-
[5]
Person image synthesis via denoising diffusion model,
A. K. Bhunia, S. Khan, H. Cholakkal, R. M. Anwer, J. Laaksonen, M. Shah, and F. S. Khan, “Person image synthesis via denoising diffusion model,” inCVPR, 2023, pp. 5968–5976
2023
-
[6]
Pose guided person image generation,
L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool, “Pose guided person image generation,” inNeurIPS, 2017
2017
-
[7]
Progressive pose atten- tion transfer for person image generation,
Z. Zhu, T. Huang, B. Shi, M. Yu, B. Wang, and X. Bai, “Progressive pose atten- tion transfer for person image generation,” inCVPR, 2019, pp. 2347–2356. 25
2019
-
[8]
Nerf: Representing scenes as neural radiance fields for view synthesis,
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” inECCV, 2020
2020
-
[9]
Smpl: A skinned multi-person linear model,
M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,”ACM Transactions on Graphics, vol. 34, no. 6, 2015
2015
-
[10]
A variational u-net for conditional appear- ance and shape generation,
P. Esser, E. Sutter, and B. Ommer, “A variational u-net for conditional appear- ance and shape generation,” inCVPR, 2018, pp. 8857–8866
2018
-
[11]
Pot-gan: Pose trans- form gan for person image synthesis,
T. Li, W. Zhang, R. Song, Z. Li, J. Liu, X. Li, and S. Lu, “Pot-gan: Pose trans- form gan for person image synthesis,”IEEE TIP, vol. 30, pp. 7677–7688, 2021
2021
-
[12]
Pona: Pose-guided non-local attention for human pose transfer,
K. Li, J. Zhang, Y . Liu, Y .-K. Lai, and Q. Dai, “Pona: Pose-guided non-local attention for human pose transfer,”IEEE TIP, vol. 29, pp. 9584–9599, 2020
2020
-
[13]
Pose flow learning from person images for pose guided synthesis,
H. Zheng, L. Chen, C. Xu, and J. Luo, “Pose flow learning from person images for pose guided synthesis,”IEEE TIP, vol. 30, pp. 1898–1909, 2020
1909
-
[14]
Towards fine-grained human pose transfer with detail replenishing network,
L. Yang, P. Wang, C. Liu, Z. Gao, P. Ren, X. Zhang, S. Wang, S. Ma, X. Hua, and W. Gao, “Towards fine-grained human pose transfer with detail replenishing network,”IEEE TIP, vol. 30, pp. 2422–2435, 2021
2021
-
[15]
Renderpeople,
“Renderpeople,” https://renderpeople.com/3d-people, 2018
2018
-
[16]
Deephuman: 3d human recon- struction from a single image,
Z. Zheng, T. Yu, Y . Wei, Q. Dai, and Y . Liu, “Deephuman: 3d human recon- struction from a single image,” inICCV, 2019, pp. 7739–7749
2019
-
[17]
Humannerf: Free-viewpoint rendering of moving people from monocular video,
C.-Y . Weng, B. Curless, P. P. Srinivasan, J. T. Barron, and I. Kemelmacher- Shlizerman, “Humannerf: Free-viewpoint rendering of moving people from monocular video,” inCVPR, 2022, pp. 16 210–16 220
2022
-
[18]
Ani- matable neural radiance fields for modeling dynamic human bodies,
S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao, “Ani- matable neural radiance fields for modeling dynamic human bodies,” inICCV, 2021, pp. 14 314–14 323. 26
2021
-
[19]
Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,
S. Peng, Y . Zhang, Y . Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou, “Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,” inCVPR, 2021, pp. 9054–9063
2021
-
[20]
Human101: Training 100+fps human gaussians in 100s from 1 view,
M. Li, J. Tao, Z. Yang, and Y . Yang, “Human101: Training 100+fps human gaussians in 100s from 1 view,”arXiv preprint arXiv:2312.15258, 2023
-
[21]
Gauhuman: Articulated gaussian splatting from monocular human videos,
S. Hu and Z. Liu, “Gauhuman: Articulated gaussian splatting from monocular human videos,”arXiv preprint arXiv:2312.02973, 2023
-
[22]
M. Kocabas, J.-H. R. Chang, J. Gabriel, O. Tuzel, and A. Ranjan, “Hugs: Human gaussian splats,”arXiv preprint arXiv:2311.17910, 2023
-
[23]
Gaussianbody: Clothed hu- man reconstruction via 3d gaussian splatting,
M. Li, S. Yao, Z. Xie, K. Chen, and Y .-G. Jiang, “Gaussianbody: Clothed hu- man reconstruction via 3d gaussian splatting,”arXiv preprint arXiv:2401.09720, 2024
-
[24]
3d gaussian splatting for real-time radiance field rendering,
B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,”ACM Transactions on Graphics, vol. 42, no. 4, 2023
2023
-
[25]
Svad: From single image to 3d avatar via synthetic data generation with video diffusion and data augmentation,
Y . Choi, “Svad: From single image to 3d avatar via synthetic data generation with video diffusion and data augmentation,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 3137–3147
2025
-
[26]
Disco4d: Disentan- gled 4d human generation and animation from a single image,
H. E. Pang, S. Liu, Z. Cai, L. Yang, T. Zhang, and Z. Liu, “Disco4d: Disentan- gled 4d human generation and animation from a single image,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 26 331– 26 344
2025
-
[27]
Sings: Animatable single-image human gaussian splats with kinematic priors,
Y . Wu, X. Chen, W. Li, S. Jia, H. Wei, K. Feng, J. Chen, Y . Li, A. He, W. Zhang et al., “Sings: Animatable single-image human gaussian splats with kinematic priors,” inProceedings of the Computer Vision and Pattern Recognition Confer- ence, 2025, pp. 5571–5580. 27
2025
-
[28]
Deep image spatial transformation for person image generation,
Y . Ren, X. Yu, J. Chen, T. H. Li, and G. Li, “Deep image spatial transformation for person image generation,” inCVPR, 2020, pp. 7690–7699
2020
-
[29]
Pise: Person image synthesis and editing with decoupled gan,
J. Zhang, K. Li, Y .-K. Lai, and J. Yang, “Pise: Person image synthesis and editing with decoupled gan,” inCVPR, 2021, pp. 7982–7990
2021
-
[30]
Deformable gans for pose-based human image generation,
A. Siarohin, E. Sangineto, S. Lathuiliere, and N. Sebe, “Deformable gans for pose-based human image generation,” inCVPR, 2018, pp. 3408–3416
2018
-
[31]
Dense intrinsic appearance flow for human pose transfer,
Y . Li, C. Huang, and C. C. Loy, “Dense intrinsic appearance flow for human pose transfer,” inCVPR, 2019, pp. 3693–3702
2019
-
[32]
Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis,
W. Liu, Z. Piao, J. Min, W. Luo, L. Ma, and S. Gao, “Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis,” inICCV, 2019, pp. 5904–5913
2019
-
[34]
Animate anyone: Consistent and controllable image-to-video synthesis for character animation,
L. Hu, “Animate anyone: Consistent and controllable image-to-video synthesis for character animation,” inCVPR, 2024, pp. 8153–8163
2024
-
[35]
Conditional Generative Adversarial Nets
M. Mirza and S. Osindero, “Conditional generative adversarial nets,”arXiv preprint arXiv:1411.1784, 2014
work page internal anchor Pith review Pith/arXiv arXiv 2014
-
[36]
Liquid warping gan with at- tention: A unified framework for human image synthesis,
W. Liu, Z. Piao, Z. Tu, W. Luo, L. Ma, and S. Gao, “Liquid warping gan with at- tention: A unified framework for human image synthesis,”IEEE TPAMI, vol. 44, no. 9, pp. 5115–5133, 2022
2022
-
[37]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, 2020, pp. 6840–6851
2020
-
[38]
Diffusion models beat gans on image synthesis,
P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” in NeurIPS, 2021, pp. 8780–8794. 28
2021
-
[39]
Glide: Towards photorealistic image generation and editing with text-guided diffusion models,
A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” inICML, 2022, pp. 16 784–16 804
2022
-
[40]
Classifier-Free Diffusion Guidance
J. Ho and T. Salimans, “Classifier-free diffusion guidance,”arXiv preprint arXiv:2207.12598, 2022
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[41]
Disco: Disentangled control for realistic human dance generation,
T. Wang, L. Li, K. Lin, Y . Zhai, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang, “Disco: Disentangled control for realistic human dance generation,” in CVPR, 2024, pp. 9326–9336
2024
-
[42]
Magicanimate: Temporally consistent human image animation using dif- fusion model,
Z. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. Z. Shou, “Magicanimate: Temporally consistent human image animation using dif- fusion model,” inCVPR, 2024, pp. 1481–1490
2024
-
[43]
Towards multiple character image animation through en- hancing implicit decoupling,
J. Xue, H. Wang, Q. Tian, Y . Ma, A. Wang, Z. Zhao, S. Min, W. Zhao, K. Zhang, H.-Y . Shumet al., “Towards multiple character image animation through en- hancing implicit decoupling,” inICLR, 2025
2025
-
[44]
360-degree human video gener- ation with 4d diffusion transformer,
R. Shao, Y . Pang, Z. Zheng, J. Sun, and Y . Liu, “360-degree human video gener- ation with 4d diffusion transformer,”ACM TOG, vol. 43, no. 6, pp. 1–13, 2024
2024
-
[45]
Champ: Controllable and consistent human image animation with 3d paramet- ric guidance,
S. Zhu, J. L. Chen, Z. Dai, Z. Dong, Y . Xu, X. Cao, Y . Yao, H. Zhu, and S. Zhu, “Champ: Controllable and consistent human image animation with 3d paramet- ric guidance,” inECCV. Springer, 2024, pp. 145–162
2024
-
[46]
Improved denoising diffusion probabilistic mod- els,
A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic mod- els,” inICML, 2021, pp. 8162–8171
2021
-
[47]
Learning to count everything,
V . Ranjan, U. Sharma, T. Nguyen, and M. Hoai, “Learning to count everything,” inCVPR, 2021, pp. 3394–3403
2021
-
[48]
One-shot generative domain adaptation,
C. Yang, Y . Shen, Z. Zhang, Y . Xu, J. Zhu, Z. Wu, and B. Zhou, “One-shot generative domain adaptation,” inICCV, 2023, pp. 7733–7742. 29
2023
-
[49]
Test-time domain adaptation by learning domain-aware batch normalization,
Y . Wu, Z. Chi, Y . Wang, K. N. Plataniotis, and S. Feng, “Test-time domain adaptation by learning domain-aware batch normalization,” inAAAI, vol. 38, no. 14, 2024, pp. 15 961–15 969
2024
-
[50]
Image quality assessment through fsim, ssim, mse and psnr—a comparative study,
U. Sara, M. Akter, and M. S. Uddin, “Image quality assessment through fsim, ssim, mse and psnr—a comparative study,”Journal of Computer and Communi- cations, vol. 7, no. 3, pp. 8–18, 2019
2019
-
[51]
Image quality as- sessment: from error visibility to structural similarity,
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality as- sessment: from error visibility to structural similarity,”IEEE TIP, vol. 13, no. 4, pp. 600–612, 2004
2004
-
[52]
The unreasonable effectiveness of deep features as a perceptual metric,
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” inCVPR, 2018, pp. 586– 595
2018
-
[53]
Controllable person image synthesis with pose-constrained latent diffusion,
X. Han, X. Zhu, J. Deng, Y .-Z. Song, and T. Xiang, “Controllable person image synthesis with pose-constrained latent diffusion,” inICCV, 2023, pp. 22 768– 22 777
2023
-
[54]
Humans in 4d: Reconstructing and tracking humans with transformers,
S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik, “Humans in 4d: Reconstructing and tracking humans with transformers,” inICCV, 2023, pp. 14 783–14 794
2023
-
[55]
Human performance modeling and rendering via neural animated mesh,
F. Zhao, Y . Jiang, K. Yao, J. Zhang, L. Wang, H. Dai, Y . Zhong, Y . Zhang, M. Wu, L. Xuet al., “Human performance modeling and rendering via neural animated mesh,”ACM TOG, vol. 41, no. 6, pp. 1–17, 2022. 30
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.