Facial Appearance Capture at Home with Patch-Level Reflectance Prior

Feng Xu; Junfeng Lyu; Kuan Sheng; Lan Xu; Minghao Que; Qixuan Zhang; Yuxuan Han

REVIEW 3 major objections 5 minor 3 references

Facial Appearance Capture at Home with Patch-Level Reflectance Prior

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A smartphone video, guided by a light-stage-trained diffusion prior, produces studio-quality facial reflectance maps.

desk verdict Patch-level diffusion prior is a real contribution, but the 'large margin' claim rests on thin quantitative evidence and an acknowledged normal-bending failure mode. read the letter →

arxiv 2506.03478 v1 pith:VH36FPRS submitted 2025-06-04 cs.GR cs.CV

Yuxuan Han , Junfeng Lyu , Kuan Sheng , Minghao Que , Qixuan Zhang , Lan Xu , Feng Xu This is my paper

classification cs.GRcs.CV

keywords facialappearancecapturereflectanceestimationdiffusionpriorposteriorsamplingLightStagesmartphoneinverserenderingpatch-level

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the quality gap between smartphone-based facial appearance capture and studio Light Stage capture can be largely closed by solving for reflectance maps inside the data distribution of high-quality scans. The proposed pipeline keeps the simple capture setup of a co-located smartphone and flashlight video in a dim room, then reconstructs 4K diffuse albedo, specular albedo, and normal maps with the help of a diffusion prior trained on patches from Light Stage data. To use that patch-level prior at full resolution, the authors develop a patch-level posterior sampling scheme that blends overlapping patches into seamless maps while matching the captured images. If the claim holds, everyday users could obtain relightable facial assets comparable to studio output, which matters because such assets currently require expensive, specialized capture equipment.

What carries the argument

The load-bearing mechanism is a patch-level conditional diffusion prior over a 9-channel UV-space representation: three channels of diffuse albedo, one of specular albedo, three of normal, and two of UV coordinates. Trained on 256-by-256 crops from 4K Light Stage scans, the prior models the joint distribution of high-quality reflectance while the UV coordinate patch acts as a positional condition, so the model knows where on the face a patch sits. To sample full-resolution maps, the authors split the map into overlapped patches, run independent DPS updates that push each patch toward photometric agreement with the captured views, and then average the overlapped regions with binary masks in the style of Tiled Diffusion, preventing adjacent patches from settling into different local minima.

What would settle it

Take a subject for whom a ground-truth polarization-based Light Stage scan exists, run the pipeline with deliberately shifted camera calibration, and compare the recovered normal maps against the ground truth. If the normal maps bend to absorb the calibration error while the rendered images stay photo-consistent, the method's reflectance quality is geometry-bounded rather than purely prior-driven; conversely, stable normals under perturbed calibration would support the paper's claim that the prior is doing the work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that ill-posedness in low-cost facial appearance capture can be compensated by a learned prior over studio-quality reflectance: specifically, a diffusion model trained on random 256-by-256 crops of 4K Light Stage diffuse albedo, specular albedo, and normal maps, conditioned on the cropped UV coordinate map. During reconstruction, geometry and lighting recovered by a prior method are frozen, and only the reflectance maps are optimized, with diffusion posterior sampling steering the generation toward reflectance that both lies in the prior distribution and reproduces the captured images under a differentiable renderer. The paper further argues that the UV-coordinate conditioning preserves global position information lost in cropping, and that alternating per-patch DPS updates with a Tiled-Diffusion blend keeps the full-resolution result seamless. The authors claim their experiments show reconstructed reflectance maps and relighting images are substantially better than prior low-cost methods and close much of the gap to a polarization-based Light Stage.

Load-bearing premise

The whole pipeline rests on the assumption that the frozen geometry and lighting estimates from the initial reconstruction are accurate; if they are not, the diffusion prior will quietly bend the normal map to absorb the error, so the final reflectance quality is bounded by the starting reconstruction's accuracy.

Editorial extensions

If this is right

Relightable facial assets with fine details such as pores, lip structure, and skin blemishes can be produced from a smartphone-and-flashlight video instead of a studio rig.
Diffuse and specular components can be disentangled without polarization filters, because the prior supplies the separation that physical filters would otherwise provide.
The patch-level training recipe makes a generative prior feasible for ultra-high-resolution, small-scale datasets, not just faces.
Reconstructed 4K maps export onto a standard facial topology, so the output plugs directly into existing graphics pipelines for animation and full-head avatar construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the authors' admission that the normal map bends to absorb camera-calibration and geometry errors implies that jointly optimizing poses and geometry during posterior sampling could sharpen the final reflectance maps; this is a direct next step the paper only sketches.
Editorial inference: because the prior is patch-based, the same 256-by-256 training-and-tiled-sampling scheme could transfer to other scarce high-resolution material datasets, such as full-body skin or fabric BRDF scans, if a dense enough patch distribution can be obtained.
Editorial inference: the roughly 8-hour runtime and the authors' coarse-to-fine suggestion point to a practical test: running the same optimization at lower resolution first and refining only selected regions could cut cost without losing the measured quality.
Editorial inference: the acknowledged failure on bushy beards suggests the 'everyday user' claims are bounded by the demographic content of the training scans; a more diverse Light Stage dataset, or finetuning on synthetic bearded scans, would directly test whether the gap closes beyond the paper's subjects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my take on arXiv:2506.03478.

The patch-level reflectance prior is the real contribution. Training a diffusion model on Light Stage scans at 256x256 patches, conditioned on UV coordinates, is a sensible answer to the ultra-high-res, small-data problem. The patch-level DPS sampler with tiled-diffusion blending also works: the ablation shows that without overlap blending you get seams, and the proposed alternating scheme produces seamless 4K maps. That is a legitimate methodological step beyond GANtlitz and DPI.

What the paper does well is the system integration and the qualitative story. The relit results look plausible, and the method handles subjects outside the training distribution, including a bearded subject when trained on DreamFace data. The authors also clearly document runtime, memory constraints, and they acknowledge the fixed roughness map and the small training set.

The soft spot is the evidence for the headline claim. The abstract says the method 'closes the quality gap by a large margin.' Table 1 shows the opposite: averaged over 8 subjects, PSNR improves by 0.37 dB, SSIM drops by 0.008, LPIPS drops by 0.002. Those are at best marginal, and the SSIM drop is a regression. The qualitative images look better, but the numbers don't capture that. The Light Stage comparison is only two subjects, purely qualitative, and it uses a degraded Light Stage without specular albedo. No metrics are reported on normal accuracy or relighting under novel lighting.

The more serious issue is the frozen geometry and lighting from CoRA. The stress test is right: the photometric loss can be satisfied by bending the normal map, and the authors explicitly acknowledge this in Sec 6.6. That means the tangent-space normal map might not be a physically correct surface-orientation asset. It can reproduce the training views while failing under new lighting. The paper does not provide any quantitative evaluation of normal error against a ground-truth Light Stage scan. For a method whose selling point is 'production-grade digital doubles,' that's a load-bearing gap.

All that said, this is not a sloppy paper. The limitations are stated honestly, the design choices are ablated, and the method is a real step toward practical high-quality face capture. It just doesn't yet prove the 'large margin' claim. The right fix is more evidence: quantitative normal error and relighting comparisons on more subjects, ideally with a full Light Stage that includes specular albedo. If the authors can provide that, the claim would be credible.

Who is this for? Anyone working on face appearance capture or diffusion priors for inverse rendering. It's a solid systems paper that deserves a serious referee. I'd send it to review with major revisions, but I would not desk-reject it.

Referee Report

3 major / 5 minor

Summary. The paper proposes a low-cost facial appearance capture pipeline that takes a single co-located smartphone and flashlight video recorded in a dim room as input. It first trains a patch-level diffusion prior on 48 Light Stage scans, conditioning on UV coordinate maps, to model the joint distribution of diffuse albedo, specular albedo, and normal in ICT UV space. During reconstruction, the method reuses CoRA's reconstructed geometry and lighting, then uses a patch-level diffusion posterior sampling (DPS) technique with tiled-diffusion blending to estimate seamless 4K reflectance maps that satisfy a photometric loss. The central claim is that this approach closes the quality gap between low-cost smartphone capture and studio Light Stage capture by a large margin.

Significance. If the central claim were fully supported, the paper would be a significant step toward democratizing high-quality facial appearance capture. The patch-level diffusion prior with UV-coordinate conditioning is a thoughtful way to learn from a small number of ultra-high-resolution Light Stage scans, and the patch-level DPS sampler with overlapped tiling is a practical memory-efficient solution. The paper includes useful ablations of its key design choices and promises code release, which are commendable. However, the quantitative evidence presented in the main text is too weak and partly contradictory to support the headline 'large margin' claim, and the physical correctness of the reconstructed normal map is not adequately verified.

major comments (3)

[§6.2.1, Table 1] The abstract and Section 1 claim that the method 'closes the quality gap ... by a large margin,' but the quantitative comparison in Table 1 does not support this. Across 8 subjects the average PSNR gain is only +0.37 dB, average SSIM drops from 0.9595 to 0.9517, and average LPIPS improves by only 0.0022; per-subject SSIM is lower in all 8 rows, and PSNR is lower for Asian M3 and African M. These are small and mixed effects, not a large margin. Please either report metrics that directly measure reflectance quality (e.g., synthetic ground-truth normal angular error, albedo error, and novel-light relighting error) or revise the 'large margin' claim.
[§6.2.2, §5.2, §6.6] The claim that the method approaches Light Stage quality rests on the reconstructed tangent-space normal map being physically correct. However, Section 5.2 freezes CoRA's geometry and lighting and optimizes only reflectance, and Section 6.6 acknowledges that the method 'tends to bend the normal to compensate for the error in camera calibration and geometry reconstruction.' Thus the photometric loss in Eq. (12) can be satisfied by a bent normal that is not a valid surface-orientation asset and that contaminates diffuse/specular disentanglement. The only evidence against this is the qualitative two-subject comparison in Figures 8-9, which reports no metrics and does not compare specular albedo. Please add quantitative normal and albedo errors against the Light Stage scans, or synthetic ground-truth experiments, and evaluate relighting under lighting conditions not used in reconstruction.
[§6.3.2, Table 2] The choice of guidance strength is not quantitatively justified. In Table 2, the row for ζ'_t=5 achieves PSNR 33.09 and LPIPS 0.065, both better than the selected configuration 'Ours' (PSNR 31.87, LPIPS 0.070), with SSIM 0.9526 versus 0.9545. Since Figure 14 is qualitative, the text's statement that ζ'_t=1 'balances quality and fidelity' is not supported by the reported metrics. Please report the metric curves over ζ'_t or explain why the qualitative criterion should dominate.

minor comments (5)

[§6.6] The subsection title 'Affects of the Data-Driven Prior' should be 'Effects of the Data-Driven Prior.'
[§4.1, §6.1.3, §6.4] The training set is ethnically imbalanced (33 of 48 scans are Caucasian), while the test set is 5 Asian, 2 Caucasian, and 1 African American; the generalization claim in Section 6.4 is based on only 8 subjects and no per-group quantitative metrics. Please provide per-ethnic-group numbers or a larger test set to support the generalization statement.
[§6.2.1] The sentence 'we conduct quantitative experiments on synthetic data in our supplementary material' should be either moved into the main paper or the supplementary should be referenced with concrete section/table numbers, since the main text otherwise lacks quantitative evidence for the reflectance-quality claim.
[§6.2.2] The Light Stage used in the comparison is a degraded implementation that cannot capture specular albedo; while this is stated in the text, the corresponding caveat should also appear in the figure captions for Figures 8 and 9 to avoid overstating the comparison.
[Figure 12] The ablation figure caption is unclear about which rows correspond to which baseline; please label each row explicitly or reference the exact configuration in the caption text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reflectance estimate is tied to captured images by a photometric loss, and the prior is trained on independent Light Stage scans; the CoRA self-citation is a pipeline component, not a proof input.

full rationale

The claimed derivation chain is self-contained. The patch-level diffusion prior (Section 4) is trained on 48 purchased 3D Scanstore Light Stage scans, which are independent of the test subjects, and the output of reconstruction is not read off from that prior: DPS sampling in Section 5.2 minimizes the photometric loss L_pho(x)=sum_i ||R(x,i)-I_i||_2^2 (Eq. 12) against the captured frames through a differentiable renderer, so the predicted diffuse albedo, specular albedo, and normal are constrained by subject-specific observations. The paper explicitly discards CoRA's reflectance output ('We only use the reconstructed geometry and scene lighting while discarding the reflectance map because we will re-estimate it later'), so the final maps are not CoRA's outputs renamed. The dependence on CoRA [Han et al. 2024] for geometry and lighting is a shared-component choice from a separately published system, not an unverified self-citation invoked to force the conclusion; even if CoRA's errors propagate (as the acknowledged normal-bending failure mode in Section 6.6 suggests), that is a robustness and validation concern, not a circular reduction. The two-subject Light Stage comparison is qualitative and small-scale, but it is an external benchmark rather than an input to the method. No equation in the paper defines the prediction in terms of the fitted quantity or makes the claimed result true by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method rests on a small commercial studio dataset (48 scans), a frozen geometry and lighting pipeline from the authors' earlier CoRA work, and several hand-selected hyperparameters. The fixed roughness map is an unstated input. No new physical entities are introduced; the diffusion prior and sampling technique are algorithmic constructs, not new forces, particles, or conserved quantities.

free parameters (6)

Guidance step size scaling zeta'_t = 1.0
Hand-tuned to balance photometric fidelity and prior quality; ablations in Section 6.3.2 and Table 2 show the trade-off.
Non-overlapped patch size p = 448
Chosen to fit 24G GPU memory with padding; larger patch sizes speed up computation but reduce overlap consistency (Section 6.3.2).
Padding size p_pad = 64
Overlap width for tiled diffusion; small padding causes seams, larger padding increases runtime (Section 6.3.2).
Number of sampling steps T = 1000
Standard DPS setting; no ablation is reported for this value.
Number of views V = 20
Uniformly sampled from the captured video for reflectance estimation; no ablation is reported.
Fixed roughness map = not specified
The paper fixes roughness because no roughness prior exists, but never states the value or source of this fixed map (Section 5.2).

assumptions (6)

domain assumption The capture is shadow-free and the flashlight is well modeled as a point light with inverse-square falloff plus low-order SH ambient.
This forward model is used in Eq.(7)-(9) and underlies the photometric loss.
domain assumption CoRA's reconstructed geometry and lighting are accurate enough to be fixed during reflectance optimization.
Section 5.2 freezes these; the paper later notes normals bend to compensate for calibration errors.
domain assumption 48 Light Stage scans from 3D Scanstore are representative of the general human population for facial reflectance prior learning.
The prior is trained on these scans and the paper shows degraded results for beards, which are absent from the training set (Section 6.6).
domain assumption UV coordinate conditioning fully compensates for the global context lost by cropping patches.
Central to the patch-level prior design; validated only by ablations, with no theoretical guarantee.
standard math Diffusion posterior sampling (DPS) approximates the posterior of the inverse problem sufficiently well for this task.
Adopted from Chung et al. 2022; the step size controls approximation error.
domain assumption The differentiable renderer, mipmapping, and patch-to-screen UV mapping are correct and provide reliable gradients.
Required for gradient flow in Eq.(11); implementation details are given in Section 5.2.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Facial Appearance Capture at Home with Patch-Level Reflectance Prior." pith.science (2026). https://pith.science/paper/VH36FPRS

@misc{pith2026250603478,
  author       = {Pith},
  title        = {Pith review of: Facial Appearance Capture at Home with Patch-Level Reflectance Prior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VH36FPRS}},
  note         = {Machine review of arXiv:2506.03478}
}

read the original abstract

Existing facial appearance capture methods can reconstruct plausible facial reflectance from smartphone-recorded videos. However, the reconstruction quality is still far behind the ones based on studio recordings. This paper fills the gap by developing a novel daily-used solution with a co-located smartphone and flashlight video capture setting in a dim room. To enhance the quality, our key observation is to solve facial reflectance maps within the data distribution of studio-scanned ones. Specifically, we first learn a diffusion prior over the Light Stage scans and then steer it to produce the reflectance map that best matches the captured images. We propose to train the diffusion prior at the patch level to improve generalization ability and training stability, as current Light Stage datasets are in ultra-high resolution but limited in data size. Tailored to this prior, we propose a patch-level posterior sampling technique to sample seamless full-resolution reflectance maps from this patch-level diffusion model. Experiments demonstrate our method closes the quality gap between low-cost and studio recordings by a large margin, opening the door for everyday users to clone themselves to the digital world. Our code will be released at https://github.com/yxuhan/DoRA.

Figures

Figures reproduced from arXiv: 2506.03478 by the authors.

**Figure 1.** We propose a novel method for low-cost high-quality facial appearance capture. Given a single co-located smartphone and flashlight sequence captured [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Overview of our patch reflectance dataset. It contains quadruples [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 12.** 5 Prior-Guided Reconstruction After training the diffusion model, we use it as a prior in the facial appearance capture process. As our method takes the same input as CoRA, we first reconstruct facial geometry and scene lighting using CoRA (Section 5.1). Then, we fix the geometry and lighting while only optimizing the facial reflectance maps within the distribution modeled by our diffusion prior (Section 5.2). 5.1 L… view at source ↗

Figures from the paper (16 more)

**Figure 3.** Figure 3: Pipeline of the proposed patch-level DPS technique. At time step [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]

**Figure 4.** Figure 4: Illustration on overlapped-patch splitting. We first split the full map [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Qualitative comparison on reflectance map reconstruction of our method and CoRA [Han et al [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Qualitative comparison on face relighting of our method and CoRA [Han et al. 2024]. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Qualitative comparison on face reconstruction of our method and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Qualitative comparison on the reconstructed diffuse albedo (top) and normal (bottom) of our method and a Light Stage method [Ghosh et al [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

**Figure 9.** Figure 9: Qualitative comparison on face relighting of our method and a Light Stage method [Ghosh et al. 2011] implemented by SoulShell. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 10.** Figure 10: Qualitative comparison on face relighting of our method, HiFi3DFace [Bao et al. 2021], and MoSAR [Dib et al. 2024]. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

**Figure 11.** Figure 11: Qualitative ablation study of the network architecture on reflectance patch generation. For our method and the [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

**Figure 12.** Figure 12: Qualitative ablation study of the key design choices in our method. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

**Figure 13.** Figure 13: Qualitative ablation study of the patch size [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

**Figure 14.** Figure 14: Qualitative ablation study of the guidance weight [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

**Figure 15.** Figure 15: Although our diffusion prior is trained on only 48 Light [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]

**Figure 15.** Figure 15: Results of our method on diverse ethnic groups. Using the diffusion prior, our method can disentangle the diffuse and specular components from the [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]

**Figure 16.** Figure 16: Application of our method. By combining our method with off-the [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]

**Figure 17.** Figure 17: Limitation of our method. As the 3D Scanstore dataset does not [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[2013]

InACM SIGGRAPH 2013 Posters

Digital ira: Creating a real-time photoreal digital actor. InACM SIGGRAPH 2013 Posters. 1–1. Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The digital emily project: photoreal facial modeling and animation. InAcm siggraph 2009 courses. 1–15. Dejan Azinović, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies

work page 2013
[2021]

in-the-wild

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies.ACM Transactions on Graphics(2021). Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. 2023. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation.arXiv preprint arXiv:2302.08113 (2023). Shrisha Bharadwaj, Yufeng Zheng, Otmar Hilliges, Michael J Black, and Victoria Ferna...

arXiv 2021
[2023]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

High-res facial appearance capture from polarized smartphone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16836–16846. ACM Trans. Graph., Vol. 44, No. 4, Article . Publication date: August 2025. Facial Appearance Capture at Home with Patch-Level Reflectance Prior•15 Linchao Bao, Xiangkai Lin, Yajing Chen, H...

work page 2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2013]

InACM SIGGRAPH 2013 Posters

Digital ira: Creating a real-time photoreal digital actor. InACM SIGGRAPH 2013 Posters. 1–1. Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The digital emily project: photoreal facial modeling and animation. InAcm siggraph 2009 courses. 1–15. Dejan Azinović, Olivier Maury, Christophe Hery, Matthias Nießner, and Justus Thies

work page 2013

[2] [2021]

in-the-wild

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies.ACM Transactions on Graphics(2021). Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. 2023. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation.arXiv preprint arXiv:2302.08113 (2023). Shrisha Bharadwaj, Yufeng Zheng, Otmar Hilliges, Michael J Black, and Victoria Ferna...

arXiv 2021

[3] [2023]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

High-res facial appearance capture from polarized smartphone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16836–16846. ACM Trans. Graph., Vol. 44, No. 4, Article . Publication date: August 2025. Facial Appearance Capture at Home with Patch-Level Reflectance Prior•15 Linchao Bao, Xiangkai Lin, Yajing Chen, H...

work page 2025