Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

PRM: Photometric Stereo based Large Reconstruction Model

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read PRM shows that training a large reconstruction model on photometric stereo images—with varied materials and lighting—yields meshes with finer local detail and strong robustness to glossy or dark input appearances.

desk verdict Solid LRM variant with a real renderer-transfer problem: the photometric-stereo gains are large but measured inside the paper's own split-sum pipeline. read the letter →

arxiv 2412.07371 v1 pith:YH73F3AI submitted 2024-12-10 cs.CV cs.GR

classification cs.CVcs.GR
keywords photometricstereolargereconstructionmodelphysically-basedrendering3Dmeshalbedopredictionspecularrobustnesstriplanerepresentationsplit-sumapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes PRM, a feed-forward model that reconstructs a 3D mesh from one or a few images. Its central claim is that if the training images—both the inputs and the supervision—are rendered as photometric stereo images, with randomly varied material and lighting conditions, the model learns finer local surface detail and becomes robust to glossy, dark, or highlight-heavy input appearances. The authors argue that previous large reconstruction models trained with fixed, simple lighting entangle texture and geometry, and that replacing those images with photometric stereo images supplies the shading equations needed to pin down surface normals. If correct, the approach makes single-image 3D reconstruction more accurate on everyday objects with varied appearances, and it enables relighting and material editing from a single input.

What carries the argument

The load-bearing object is the photometric stereo image, produced online during training by combining mesh rasterization with the split-sum approximation to the rendering equation. The split-sum approximation rewrites the specular term as a precomputed BRDF integral times a mipmap-sampled lighting term, so the model can render many material/lighting combinations in real time; the mesh representation (FlexiCubes) makes this rendering differentiable and yields albedo, normal, depth, specular-light, and diffuse-light maps. These maps give extra supervision terms—especially the lighting maps, which are functions of the surface normal alone—so the network is solving the photometric stereo system of equations rather than just predicting colors.

What would settle it

Render a held-out set of objects with both the split-sum pipeline and a full Monte Carlo path tracer, then run PRM on the path-traced images: if Chamfer Distance or F-Score degrades markedly relative to split-sum renderings, the photometric cues are partly renderer artifacts. Alternatively, evaluate PRM on calibrated real captures with known surface geometry, where input appearance includes uncontrolled specularities and shadows, and compare error against the synthetic-test numbers reported in the paper.

Watch

Extended reading notes

Core claim

The paper's central claim is that photometric stereo images—rendered by varying camera pose, metallic, roughness, and environment lighting—are a better training distribution for large reconstruction models than the fixed-lighting images used by predecessors. By using these images as both input and supervision, PRM reduces the geometry/texture ambiguity that makes glossy surfaces come out bumpy or wrong, and it recovers fine local relief that albedo-only supervision misses. The authors report that on GSO, Chamfer Distance falls from 0.076 (InstantMesh) to 0.050 and F-Score@0.1 rises from 0.931 to 0.981, with comparable gains on OmniObject3D, and that the model also predicts albedo, allowing relighting and material editing. The mechanism behind the gain is that shading color and the intermediate specular and diffuse lighting maps carry photometric cues that depend only on surface normals, so supervising on them acts like photometric stereo and sharpens the geometry.

Load-bearing premise

The whole photometric-stereo benefit rests on the real-time split-sum approximation used to render both the training inputs and the supervision images: if that rendering shortcut is systematically biased, the photometric cues become renderer-specific artifacts and the robustness gains may not carry over to real photographs.

Editorial extensions

If this is right

  • Training on photometric stereo images should make a reconstruction model generalize to input photos with specular highlights, cast shadows, and dark appearances, because the model has seen those shading patterns during training.
  • Supervising on specular and diffuse lighting maps, not just final color, directly constrains surface normals and should improve fine local relief on smooth or textureless surfaces.
  • Predicting albedo rather than shading color makes the output separable into geometry and material, enabling relighting and material editing as post-hoc operations.
  • Rendering training data online with rasterization and split-sum approximation removes the offline rendering bottleneck, so the training distribution can be expanded to varied camera poses, fields of view, and material combinations cheaply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: feeding PRM a set of images of the same object under several lighting conditions should let it output normals and albedo directly, effectively turning the reconstruction model into a fast multiview photometric stereo solver—something the paper does not evaluate.
  • Because the split-sum approximation is shared between training and supervision, an internal consistency check (comparing PRM's reconstructions when trained with full Monte Carlo PBR as supervision) would tell whether the reported gains come from true photometric cues or from the network exploiting the renderer's shortcut, a comparison the paper does not run.
  • The same online rendering pipeline could be used to ablate the value of other appearance variations—camera field-of-view, environment map intensity, or cast shadows—to isolate which photometric cue contributes most to the detail improvement.
  • The model's robustness to background clutter is limited by its white-background training; combining photometric stereo rendering with background augmentation is a natural direction the paper mentions only in passing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes PRM, a feed-forward large reconstruction model that reconstructs meshes from single or multi-view images by leveraging photometric stereo images rendered with randomly varied materials and lighting under a real-time split-sum PBR approximation. The model uses FlexiCubes as an explicit mesh representation, enabling differentiable PBR to render shading variables such as albedo, specular/diffuse light maps, normals, and depth for multi-term supervision (Eq. 12). Experiments on GSO and OmniObject3D report significant improvements over prior methods (e.g., GSO CD 0.050 vs 0.076 for InstantMesh; FS@0.1 0.981 vs 0.931; PSNR 25.1 vs 20.0), along with a controlled comparison in Table 2 using ground-truth multi-view inputs. The paper also demonstrates qualitative robustness on real images and supports relighting and material editing applications.

Significance. If the reported gains are reproducible and transfer beyond the paper's own rendering pipeline, PRM is a meaningful advance: it introduces a practical way to inject photometric stereo cues into large reconstruction models, with a real-time split-sum rendering pipeline that enables online data generation, and an explicit mesh representation with differentiable PBR supporting multiple photometric supervisions and downstream relighting/editing. The paper's strengths include consistent quantitative improvements across two datasets, a controlled comparison with ground-truth multi-view input (Table 2), and a systematic ablation study (Table 3). The qualitative results on real images (Figs. 13 and 18) are encouraging. However, the evaluation is currently closed-loop with respect to the rendering approximation, which tempers the strength of the central claims and requires additional validation.

major comments (3)
  1. [Sec. 3.2 (Eqs. 7-9) and Sec. 4.1] The quantitative evaluation is conducted entirely within the same rendering pipeline used for training. Photometric stereo images, the supervision maps (L_spec, L_diff, normal, etc.), and the test-set ground-truth images are all produced with the split-sum approximation (Eqs. 7-9). Consequently, the reported improvements over prior methods (e.g., CD 0.050 vs 0.076 on GSO) may reflect that PRM learns the systematic errors of this particular approximation rather than physically generalizable photometric cues. The paper does not compare the split-sum output against Monte Carlo path tracing, and real-image results appear only qualitatively (Figs. 13 and 18). To substantiate the claim of robustness to real appearances, I request either a quantitative evaluation on a test set rendered with an independent path tracer, or a real-capture benchmark with geometric metrics, or at minimum an analysis of the split-sum approximation error on representative materials.
  2. [Sec. 4.1, Evaluation Protocol] The protocol for 2D metrics (PSNR/SSIM/LPIPS) is under-specified. The authors state that novel views are rendered from the reconstructed meshes and compared to ground truth, but they do not state which renderer is used for this step. If PRM's predicted meshes are rendered with the same split-sum PBR as its training (which appears to be the case from Sec. 3.3), while competing methods are rendered with a different standard renderer (e.g., Blender Cycles), the comparison is not fair and the PSNR gains could be inflated. The authors must specify a single common rendering pipeline for all evaluated methods, or otherwise demonstrate that the results are insensitive to the renderer choice.
  3. [Table 3 (Appendix A.3)] The ablation study is trained on a reduced subset of 10k objects, whereas the full model is trained on a larger filtered Objaverse set (size not disclosed). As a result, the 'Full model' row in Table 3 reports CD 0.066, which is notably worse than the main evaluation's CD 0.050 for PRM. The relative order of ablations is informative, but the quantitative magnitudes do not directly transfer to the full-scale model. Please state this limitation clearly, and if feasible, provide full-scale ablations for at least the most important components (e.g., w/o lighting supervision).
minor comments (6)
  1. [Sec. 3.1] There is a typo: 'reconstrcut' should be 'reconstruct'.
  2. [Sec. 3.3] The text says 'mean squaree error loss'; this should be 'mean squared error loss'.
  3. [Figure 5 caption] The caption uses 'w/o PRB' where it should be 'w/o PBR'.
  4. [Appendix A.9] The phrase 'while background' appears twice and should be 'white background'.
  5. [Sec. 3.2, Discussion] The photometric-stereo formulation assumes known lighting, metallic, and roughness per view, but at inference the network only receives RGB images. The paper should clarify that the model does not perform classical photometric stereo at inference; rather, it learns a mapping from multi-view appearance to shape and albedo, with the PS formalism serving as a training-data design principle.
  6. [Tables 1-3] No error bars or statistical significance tests are reported for any quantitative result. Given the 300-object evaluation set, reporting results over a few random evaluation subsets or training runs would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: photometric-stereo supervision and external benchmarks keep the evaluation non-tautological.

full rationale

PRM's derivation is a supervised-learning pipeline rather than a closed-form derivation, and none of its reported results reduce to its own inputs by construction. Training renders ground-truth Objaverse meshes with the split-sum PBR approximation (Eqs. 7–9) and optimizes the loss in Eq. (12) against those renderer outputs; this is standard training supervision, not a fitted parameter renamed as a prediction. The quantitative evaluation on GSO and OmniObject3D compares Chamfer Distance and F-Score against external mesh geometry, and the 2D metrics compare rendered novel views to ground-truth views; these benchmarks do not use trainable constants, so the numeric gains over InstantMesh and other baselines are not forced by the model's definition. The photometric-stereo equations (Eqs. 3–6) are classical external results used as motivation, not author-specific uniqueness theorems, and the paper does not invoke any self-citation to forbid alternative designs or to define its representation. The few self-citations (Ref-NeuS, X-ray, FlexGen) appear in related-work or motivation contexts and are not load-bearing premises of the method. The only substantive limitation is that training, supervision, and quantitative evaluation all use the same split-sum renderer, so the appearance-robustness claim is not yet quantitatively validated on Monte Carlo rendering or real captures; real-image results are qualitative. That is an external-validity and correctness risk, not circularity, because external object geometry and independent baseline models still make the benchmark non-tautological.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The photometric stereo images are rendered data, not a new ontological commitment.

free parameters (2)
  • loss weights in Eq. (12) = lambda_LPIPS=2.0, lambda_normal=0.2, lambda_depth=0.5, lambda_mask=1.0, lambda_reg=0.01
    Chosen by hand; no sensitivity analysis is reported, so it is unclear how much the reported gains depend on these values.
  • per-view consistency threshold for materials and lighting = 50 percent probability of changing materials and lighting per view
    Appendix A.4; this training-distribution choice directly shapes the robustness behavior but is selected by hand without ablating the threshold.
assumptions (4)
  • domain assumption Split-sum approximation gives a sufficiently accurate and differentiable approximation to Cook-Torrance specular reflection.
    Invoked in Eqs. (7) to (9), Section 3.2; the entire online rendering pipeline and its supervision rely on this approximation.
  • domain assumption The sampled Polyhaven environment maps and 121 metallic and roughness combinations cover the appearance variation needed for robustness.
    Section 4.1; the robustness claim is tested on renderings drawn from this same distribution, so coverage is assumed rather than demonstrated.
  • domain assumption The pretrained multi-view diffusion model produces multi-view images consistent enough for single-image reconstruction.
    Figure 2 and Section 5 limitation 1; the single-view inference path depends on this external model, whose identity and training details are not given.
  • domain assumption FlexiCubes mesh extraction and mesh rasterization provide unbiased gradients for geometry optimization.
    Section 3.3 and Eq. (11); if the differentiable mesh and PBR gradients are biased, the multiple supervisions would not correct geometry as claimed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PRM: Photometric Stereo based Large Reconstruction Model." pith.science (2026). https://pith.science/paper/YH73F3AI

@misc{pith2026241207371,
  author       = {Pith},
  title        = {Pith review of: PRM: Photometric Stereo based Large Reconstruction Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YH73F3AI}},
  note         = {Machine review of arXiv:2412.07371}
}
read the original abstract

We propose PRM, a novel photometric stereo based large reconstruction model to reconstruct high-quality meshes with fine-grained local details. Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision, PRM renders photometric stereo images by varying materials and lighting for the purposes, which not only improves the precise local details by providing rich photometric cues but also increases the model robustness to variations in the appearance of input images. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh rasterization for online images rendering. Moreover, in employing an explicit mesh as our 3D representation, PRM ensures the application of differentiable PBR, which supports the utilization of multiple photometric supervisions and better models the specular color for high-quality geometry optimization. Our PRM leverages photometric stereo images to achieve high-quality reconstructions with fine-grained local details, even amidst sophisticated image appearances. Extensive experiments demonstrate that PRM significantly outperforms other models.

Figures

Figures reproduced from arXiv: 2412.07371 by the authors.

Figure 1
Figure 1. Top left: PRM is capable of reconstructing high-quality meshes with fine-grained local [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our framework. During training, photometric stereo images are rendered [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison on shiny objects. In contrast, we prepare photometric stereo images by varying materials and lighting. A naive solu￾tion is to prepare these images offline, as with previous methods, but this approach poses significant challenges due to the infinite number of potential combinations of materials and lighting. Moreover, rendering high-quality images requires large sample counts, making traditional data prep… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons with state-of-the art methods and ground truth for single-view [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Shading color offers significant [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Ablation study of the effect of changing materials during training. Albedo supervision only. To avoid the interference caused by specular color on the surface, an intuitive approach is to directly use albedo instead of shading color for supervision. However, this metho…
Figure 9
Figure 9. Figure 9: The details of network architecture. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 18
Figure 18. Figure 18: A.8 THE EFFECT OF THE NUMBER OF CAMERA VIEWS We demonstrate the importance of varying camera poses for rendering multi-view photometric stereo images as input. The number of input views is increased from 1 to 8. The qualitative re￾sults are illustrated in [PITH_FULL_…
Figure 10
Figure 10. Figure 10: Examples of rendered photometric stereo images, along with specular, diffuse lighting [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Application visualization. We show relighting and materials editing here. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Comparison with InstantMesh when changing FOVs and camera radius: PRM demon [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Single view reconstruction results using our method on input images with extreme con [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: The effect of the number of input views. More views lead to better reconstruction result. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Illustration of a failure case. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: DepthAnythingV2 can estimate correct depth for image that lacks depth information, [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Visualization of more results of single view to 3D task. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: PRM can handle objects with spatially-varying materials for both synthetic and real [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Our method fails to handle images with natural background since we takes images with [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Hybrid Neural-Microfacet BRDF Model for Real-Time Rendering

    cs.GR 2026-08 conditional novelty 6.0 of 10

    A hybrid BRDF model, combining a GGX analytical term with a tiny learned residual and gating network, fits measured materials more accurately than fully neural models at equal memory cost.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [3]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

  2. [5]

    Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching

    Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11284,

  3. [6]

    Uncalibrated photo- metric stereo under natural illumination

    Zhipeng Mo, Boxin Shi, Feng Lu, Sai-Kit Yeung, and Yasuyuki Matsushita. Uncalibrated photo- metric stereo under natural illumination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

  4. [9]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,

  5. [10]

    Lgm: Large multi-view gaussian model for high-resolution 3d content creation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054,

  6. [14]

    Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models

    Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024a. Xinli Xu, Wenhang Ge, Jiantao Lin, Jiawei Feng, Lie Xu, HanFeng Zhao, Shunsi Zhang, and Ying- Cong Chen. Flexgen: Flexible multi-vi...

  7. [15]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.09414,

  8. [16]

    Clay: A controllable large-scale generative model for creating high-quality 3d assets

    Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. arXiv preprint arXiv:2406.13897,

Show all 17 references
  1. [17]

    3.1 we introduce the D, F and G term of the specular component of BRDF property

    13 PRM: Photometric Stereo based Large Reconstruction Model A A PPENDIX A.1 BRDF P ARAMETERIZATION In Sec. 3.1 we introduce the D, F and G term of the specular component of BRDF property. We implement the Cook-Torrance BRDF model (Cook & Torrance, 1982). The basic specular alb...

  2. [1987]

    Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model

    Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint arXiv:2311.06214,

  3. [2013]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988,

  4. [2018]

    Dual marching cubes

    Gregory M Nielson. Dual marching cubes. In IEEE visualization 2004,

  5. [2020]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400,

  6. [2021]

    Crm: Single image to 3d textured mesh with convolutional reconstruction model

    Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.05034,

  7. [2022]

    Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction

    Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689,

  8. [2023]

    Google scanned objects: A high-quality dataset of 3d scanned household items

    Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA),

  9. [2024]

    Triposr: Fast 3d object reconstruction from a single image

    12 PRM: Photometric Stereo based Large Reconstruction Model Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv p...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.